Join the SF MySQL Meetup for our April meetup where Bret Taylor from FriendFeed talks about how they store their data in a "schema-less" system on top of MySQL. FriendFeed stores over 250 million entries and a bunch of other data, from comments and "likes" to friend lists.

As the database has grown, they have tried to iteratively deal with the scaling issues that come with rapid growth. They did the typical things, like using read slaves and memcache to increase read throughput and sharding our database to improve write throughput. However, as FriendFeed grew, scaling their existing features to accommodate more traffic turned out to be much less of an issue than adding new features.

In particular, making schema changes or adding indexes to a database with more than 10 – 20 million rows completely locks the database for hours at a time. After some deliberation, FriendFeed decided to implement a "schema-less" storage system on top of MySQL rather than use a completely new storage system. This talk attempts to describe the high-level details of the system.

Bret Tayler is one of the founders of FriendFeed.com. Before starting FriendFeed, he was a Group Product Manager at Google, responsible for Google’s developer products. Before that, he launched Google Maps and Google Local, and before that he worked on Google’s crawling and indexing infrastructure.

So if you’re in the SF area on Monday Feb 2nd, 2009, stop by the CBSi building and listen in.

Tungsten Replicator is a new open source project for database replication. It offers MySQL to Oracle replication, robust handling of master failures, extremely flexible topologies, and an extensible architecture that allows you to add your own replication features if you don’t find them there already. I’ll describe why MySQL needs more replication, show the product architecture, describe some of the problems it solves, and demonstrate the replicator in action. Finally, we’ll play “stump the chump” where you can present your replication problem and I’ll describe how our replicator handles it. Whether you have a tricky replication problem or are just curious about the subject, please join us. This is a very technical presentation that should be of interest to all true MySQL fans.

More information on Tungsten Replicator: http://www.continuent.com/community/tungsten-replicator

Peter Zaitsev will be giving his excellent presentation “Scaling MySQL powered Web Sites by Sharding and Replication” to the upcoming SF MySQL 6:00pm on November 3rd @ the CBS Interactive (formerly CNET Networks) building in SOMA San Francisco (235 2nd St).

RSVP here: http://mysql.meetup.com/30/calendar/8912109/

Description from the 2008 Velocity conference:
When your application grows beyond capacity of a single MySQL server there are few ways to scale, with most typical being Replication and Sharding.

In this presentation you will learn how, depending on your application performance scalability and high availability goals, you can choose what proper scaling strategy will be good for you.

We also will speak about scalability limitations of replication of Replication and Sharding, implementation complexities, and aspects of operation.

Peter Zaitsev’s bio:
MySQL Performance Blog

Peter helped many companies ranging from one man startups to Fortune 500 companies with their Performance and Scaling problems on various stages starting from architecture design and down to queries and schema optimization.

Over cause of the years Peter has spoken on many International conferences focused on MySQL, Open Source, Databases or High Performance Web applications.

Before co-founding Percona Peter worked for MySQL AB as head of High Performance Group being involved in Support, Consulting, Development and working with vendors helping to optimize their software or hardware to perform well with MySQL.

Peter also was involved in number of Web Startups in the roles ranging from CTO to Consultant and has a lot of experience in MySQL and Web Applications Operations, Deployment, Quality Assurance and Development.
Please join us!

Map to CNET Networks, Inc.
When you arrive, please look for the PHP/MySQL Meetup sign pointing you to the conference room.

Look forward to seeing you there!

Must Have:

• Expert PHP skills, bonus for RoR
• 5-7 years of real world experience engineering web solutions for LAMP stacks
• Strong relational DB knowledge (table optimization, query optimization)
• Ability to prioritize and lead multiple simultaneous projects effectively
• Excellent understanding of OO software engineering concepts
• Ability to arhitect solid, scalable, extensible solutions for SE needs
• Excited by working on high-level design and architecture
• Comfortable mentoring junior engineers
• Expereince with formal release processes
• Strong verbal, written, analytical and communication skills

Bonus for:

• Admin experience with MySQL, Apache, SVN
• Background developing in a more structured language (C, C++, Java, etc)
• At least 3 years experience in high-traffic/high-availability web sites
• Strong UI experience with javaScript and AJAX using mootools or other JS Frameworks
• Experience with following technologies: Memcached, Smarty, SOLR, Lucene, Zend Framework

If you think you have what it takes, head over to our Jobs site and apply: https://careers.cnetnetworks.com/joblist.html

If you want to discuss the job in more detail first, please don’t hesitate to send me an
at michael[
]tougeron[
]cnet[
]com.

MySQL Proxy has a c-based event driven core that has command line support, event handling and a Lua-based scripting layer. The proxy is loading into MySQL via the plugin interface. With the community edition, you are responsible for writing your own scripts. But with MySQL Enterprise, it will come with several pre-built scripts that add some enhanced functionality.

The most basic use of MySQL Proxy is to rewrite or alter your queries. If you have a costly query that is killing your db, but you’re not sure where it is coming from, you can write a Lua script to either reject the query or to alter it and make it more manageable. It stops the dba from having to wait for a developer to update code and/or deploy the fix.

Another trick you can do with MySQL Proxy is to create new SQL commands. While this is probably not going to be a great idea, you can see a few interesting examples at http://thenoyes.com/littlenoise/?p=63. It is a good way to expand on functionality at the lowest level. Or to lose/forget where you put it when you switch to another db.

One of the features MySQL was thinking of when they built MySQL Proxy is the ability to manage your queries at the database level. Imagine tracking query quotas, by ip, user, db, etc., then rejecting queries automatically if over quota. Or perhaps to simulate a db outage or transaction failure when testing code to ensure properly fail-over or error handling.

One of the default Lua scripts that comes with MySQL Proxy is a basic load balancer for the db. It rotates the connections round-robin betweeen the MySQL servers. Since it is in the database layer, it can be aware of slag lag and adjust the rotation accordingly. Since it is a Lua script, you can customize it to work how you want it to.

In MySQL Enterprise, this script is on steroids and does so much more. The load balancer automatically checks the slave state, including lag, # of connections, last error, etc., and kicks the server from rotation if it finds something “wrong” with the slave. This functionality will be integrated with MySQL Enterprise Monitor so that you will have an easy view into how the load balancer is working.

Also for Enterprise, the MySQL team is working on an automated sharding algorithm to go with MySQL 5.1′s table partitioning. I’m not sure how easy that will be to implement, but if they are able to pull it off, I’ll be mighty happy. At least for basic partitioning or sharding, like with a user table, you could do it through MySQL Proxy without having to worry about missing an update someplace in your code. If you combine this with logging, you can update your code at your leisure and easily find the queries you missed without having to worry about breaking the site.

Other planned improvements include being able to run parallel queries, two-phased commits and syncronyous replication. Since the proxy is not the database engine, it can use multiple cores on a server and assign out queries and other processes in ways that the db engine you are using cannot.

This is still very new technology/product for MySQL, but I’m looking forward to what comes out of it. There is a lot of possibility in it, as well as the potential for bloat. Personally, I think it will end up back as MySQL core so that the engines can take advantage of some of the Proxy abilities.

Maria was designed to be a crash-safe replacement for MyISAM. Maria supports the same row formats and features as MyISAM. This means it can be fully compatible with MyISAM by simply running it in non-transactional control. It will be ACID compliant and have multi-version concurrency control (MVCC).

Version 1.0 of Maria was released in January as a branch of MySQL. It will continue to be a branch until version 2.0 where it will hopefully be part of MySQL 6.  Customized documentation is also available on http://dev.mysql.com/doc/refman/5.1-maria/en/se-maria.html.

Version 1.5 of Maria should be released any week now. Monty had originally hoped to announce it this week at the conference, but it is not quite done yet. He says that it should be out sometime between Late April to mid-May. Starting with version 1.5, Maria will support concurrent INSERT and SELECT operation. This will give Maria a nice feature merge between InnoDB and MyISAM.

Starting with MySQL 6 in Q4 of 2008, version 2 of Maria will be fully transactional and ACID compliant. Having the crash recovery will be a great thing to have with the read speed of MyISAM. Version 3 will be optimized for higher concurrency.

With the expected small footprint and the efficient data storage model, Maria will be a good engine of choice for desktop applications or embedding in applications.

The way in which Maria is crash-safe is kind of interesting. It stores everything except table definitions into log files. While these log files can be quite large, starting size is 1GB, it creates a few cool features. With MyISAM tables, you can apply a lock to the db (or shutdown the server) and then copy the data to a new server. With Maria, you can simply copy the data and the logs to the new server. Like with MyISAM, the new server will have corrupted tables. Unlike MyISAM, Maria will then run through the logs and recover the db to where it should be. Or, if you don’t like that method, you can take last night’s backup to the new server, copy just today’s logs from the donor server and start up MySQL. Maria will take the data from the backup and then automatically apply the logs until the new server is up-to-date.

There is quite a list of features that are either already in Maria or will be by the time version 3 comes out.

• MVCC & ACID
• Equal lookup speed for all indexes
• Allows easy copying of tables and/or Maria logs
• Has instant COUNT(*) queries
• Better BLOG handling than MyISAM
• Versioned
• Crash safe
• Data stored in 8k pages for less locking and improved query performance

By using 8k pages Maria will be faster on static size indexes than MyISAM, but slower on variable length keys. Updates that are planned for version 2 or 3 will make variable length indexes just as fast.

There are a few disadvantages of Maria over MyISAm.

• No INSERT DELAYED (may or may not eventually be added)
• Fulltext indexes are not yet crash-safe (planned to be fixed in version 3)
• No external locking
• Like MyISAM it will not support RAID
• When using checksums, the data size will be slightly larger for indexes
• Storage of very small rows (<25 bytes) are not efficient with the new page format

All in all, I’m pretty excited about where they are going with Maria. While some of the databases I use are write heavy the vaste majority are primarily reads; there are only so many news stories a day. I looks like Maria will give us the stability and locking abilities of InnoDB while keeping the speed of MyISAM reads.

The new table partitioning system has several benefits for your large datasets where you are having performance issues due to the volume of data. It can reduce the seek & scan set sizes and reduce the INSERT/UPDATE transaction durations. If you are already performing UNION queries or secondary queries to find data in an archive table, this functionality may be for you.

There are four (point 5) partitioning styles available for your use:

1. Range
   • Very useful for situations where the data is accessed by date
   • Usually used when a limited number of primary partitions are needed
   • It supports sub-partitions (the .5 partition type)
2. List
   • Good for when you are grouping data in partitions out of order (e.g., SELECT 1,5,7 in partition X)
   • As the name implies, you list which data goes into which partition
   • Higher maintenance costs in many situations
   • Supports sub-partitions
3. Hash
   • Usually has a lower maintenance cost
   • Partitions the data based on the hash that you tell it
   • Good for when you have non-intelligent keys
   • Works with both a limited and a large number of Partitions
4. Key
   • Similar to Hash partitioning, but based on one or more of the fields in the primary key instead of a user-defined hash
   • If there is no primary key, then the first unique index is used
   • Does not require the key to be integer based
5. Subpartitions
   • As the name implies, a partition within a partition
   • You can only create subpartitions for Range & List
   • If needed, subpartitions can be spread across volumes

The alter table statements are fairly simplistic; “ALTER TABLE my_table PARTITION BY KEY() PARTITIONS 50″ will create 50 partitions for that table.  If you need to add more partitions, you can just run “ALTER TABLE my_table ADD PARTITION PARTITIONS 25″ and your table will now have 75 partitions.  Of course you can shrink the size of the partitions as well.  One of the nice features of the partition modification is that the data is re-allocated as appropriate.

All in all, this looks like a much better solution than the standard 10/90 tables.

The home for the MySQL Sandbox project is hosted on SourceForge.net. It is not an officially supported product which is why Giuseppe keeps it on SourceForge.

The Sandbox was designed so that you don’t have to go through the hassle, and potentially error prone, steps of setting up multiple sandbox databases. It’s very common for these manual installations to accidentally install in the same data directory (can corrupt your data) or install with the same port/socket (MySQL won’t start).

Details of installing and using after the break…

The simple installation steps for MySQL Sandbox automatically creates a new instance of MySQL with a sandboxed data directory, port, and socket. In order to use Sandbox’s express_install script you need to download and un-tar a copy of MySQL for each version you wish to install. The script by default uses $HOME/opt/mysql as its base directory so be sure to extract MySQL there. You can change the default source directory, but by being in your home directory it helps keep your instance separate from everyone else’s.

Next download and extract the MySQL Sandbox from SourceForge and run the following:
./express_install.pl {directory in $HOME/opt/mysql}
If you prefer, you can run ./express_install.pl /path/to/mysql/directory

You can setup replication or multiple instances by ./set_replication.pl and ./set_custom_many.pl respectively. You also have a variety of install options that can be found by adding –help to the commands.

One of the nice commands you can use once the sandbox is setup and running is “clear.” This command will reset the data and instance back to default so that you can start again from scratch. Running the “use” command will get you into the CLI of the appropriate instance. One thing to keep in mind is that the commands to interact with the sandboxes must be run from the installed directory. You also have access to some of the standard MySQL command line tools by running “my {sqldump|sqlbinlog|sqladmin}” (Note the space after my).

It seems like this would be a good system to use in conjunction with DBUnit. As I talked about yesterday, DBUnit can auto-load/populate data. By putting the two together you should have a very maintainable and separate testing environment.

The new format of InnoDB can be dynamically installed w/o re-linking MySQL, which should be pretty handy for people who cannot easily re-build MySQL. The InnoDB upgrade is fully compatible with existing InnoDB tables and can even be downgraded back to the old version if you need to.

Installation is a breeze and can be done by downloading, extracting the ha_innodb.so into the MySQL Server plugin directory, and then restarting the mysqld. Be sure to enable “skip_innodb” in the my.cnf so that it doesn’t conflict with the engine already in MySQL. The innodb_file_format is now Barracuda. Finally log into the cli and “INSTALL PLUGIN INNODB SONAME ‘ha_innodb.so’” and “INSTALL PLUGIN INNODB_LOCKS SONAME ‘ha_innodb.s’”

MySQL/InnoDB 5.1 rebuilds the entire table row by row when creating a new index. The InnoDB Plugin builds just the new index. DROP INDEX for secondary indexes has improved performance as well because it only changes the data dictionary. When changing the PRIMARY KEY or FOREIGN KEYs it still needs to rebuild the table; it’s faster then 5.1.

Any C/U/D style queries will wait on the index creation, but SELECTs will still go through in shared mode. Newly created indexes lack the historical version info for row (for only that index). This can affect consistent read SELECTs of older transactions.

According the benchmarks that Peter Zaitsev performed, creating a new index on a 3GB table it takes 88 minutes with version 5.1 and an amazing 8 minutes with the new InnoDB plugin.

With the new table compression you can choose the compressed page size on a per table basis. In order to use this functionality, you need to have innodb_file_per_table = 1 and innodb_file_format = barracuda. The patterns in the data will determine your compression rate. According to Heikki, it can often be >50%. Both the tables and the indexes are compressed using zlib. It now trys to do updates and inserted in small sets so that it does not need to recompress when the data is updated. Deletions do not require page recreation.If the page is frequently accessed, it will keep it uncompressed. It is important to monitoring the compress vs uncompressed rates in the INFORMATION_SCHEMA. As few as 1% unsuccessful compression operations wastes your cpu time.

The best time to use compression is when you are I/O bound. If you are cpu-bound, then it is not a good idea to use compression as it will spend even more cpu time compressing and uncompressing. As a test, you can gzip your .ibd data file and see what level of compression you get. If it does not compress to significantly less than 50% of the original size, it probably won’t be worth doing.

The new ROW_FORMAT = DYNAMIC also requires the Baricuda file format. It is meant for situations when you have a table with a lot of large data elements like BLOB or TEXT. Now InnoDB can store the data off-page.

With the INFORMATION_SCHEMA updates, you can now find out much more information about the status of InnoDB and the individual tables. You can even now easily find the blocking queries and which process id is the owner. This should be a great help to DBAs who need to manage large sets of InnoDB databases.

After the keynotes, I decided to start off with the Testing PHP/MySQL Applications with PHPUnit/DbUnit session by Sebastian Bermann. I know I should already be using PHPUnit, but honestly I’ve never gotten around to it. I’m hoping that this will give me the kick in the ass that I need to start.

The first thing that Sebastian wanted us to note is that the PHPUnit listed on SourceForget.net is not the real PHPUnit. I kind of thought that weird and confusing. You’d think by now that he could get that old project de-listed.

Installing PHPUnit is fairly simple. It is not hosted off the standard PEAR channel, but by a quick extra command you can get access to it.
pear channel-discover pear.phpunit.de
pear install phpunit/PHPUnit

By setting up a reusable testing environment with PHPUnit you’ll be able to do automatic execution of your test code. This would be good for things like over night tests of the latest code commits. If a test fails, the developer team can be immediately notified. Be sure to enforce strict separation between production and testing environments. The last thing you want to happen is to have your test cases somehow break production.

Setting up a test case can be as simple or complex as you need it to be. You setup a class for each element/object you need to test. Each test case is a member function of this class where you assert value comparisons. While the documentation examples are best, here is a very simple example:

<?php
class BankAccountTest extends PHPUnit_Framework_TestCase {

    public function testBalanceIsInitiallyZero() {
        $ba = new BankAccount;
        $this->assertEquals(0, $ba->getBalance());
    }
}
?>

Where things can now start to get interesting is when you use the database extension to PHPUnit. The DBUnit extension was ported from jUnit by Michael Lively Jr.

DBUnit uses PDO to talk to a default SQLite database. However, a database it not required as it can emulate a database in memory or via XML. It also works with MySQL, but if the tests are written in a way that is compatible with SQLite, then the tests can be faster by about 20%.

Once of the nice things about how DBUnit works is how it puts the db into a known state between each test run. This means it avoids problems with one test case corrupting the db for other tests. By using its ability to import/export data to/from XML datasets, your db tests will run reliably and accurately.

Like with PHPUnit, DBUnit has assert methods like “assertTableEquals(expected, actual)” and can use PHP arrays for comparison. This can be good for comparing records in the database. e.g., The db gets updated via a PHPunit test and you then do a DBUnit test to verify the action.

A quick example of a DBUnit test case:

<?php
class BankAccountTest extends PHPUnit_Extensions_Database_TestCase {
    protected $pdo;

    public function __construct() {
        $this->pdo = PHPUnit_Util_PDO::factory($dsn);
        BankAccount::createTable($this->pdo);
    }

    protected function getConnection() {
        return $tihs->createDefaultDBConnection($this->pdo, 'mysql');
    }

    protect function getDataSet() {
        return $this->createFlatXMLDataSet('/path/to/seed.xml');
    }
}
?>