Join the SF MySQL Meetup for our April meetup where Bret Taylor from FriendFeed talks about how they store their data in a "schema-less" system on top of MySQL. FriendFeed stores over 250 million entries and a bunch of other data, from comments and "likes" to friend lists.

As the database has grown, they have tried to iteratively deal with the scaling issues that come with rapid growth. They did the typical things, like using read slaves and memcache to increase read throughput and sharding our database to improve write throughput. However, as FriendFeed grew, scaling their existing features to accommodate more traffic turned out to be much less of an issue than adding new features.

In particular, making schema changes or adding indexes to a database with more than 10 – 20 million rows completely locks the database for hours at a time. After some deliberation, FriendFeed decided to implement a "schema-less" storage system on top of MySQL rather than use a completely new storage system. This talk attempts to describe the high-level details of the system.

Bret Tayler is one of the founders of FriendFeed.com. Before starting FriendFeed, he was a Group Product Manager at Google, responsible for Google’s developer products. Before that, he launched Google Maps and Google Local, and before that he worked on Google’s crawling and indexing infrastructure.

So if you’re in the SF area on Monday Feb 2nd, 2009, stop by the CBSi building and listen in.

Tungsten Replicator is a new open source project for database replication. It offers MySQL to Oracle replication, robust handling of master failures, extremely flexible topologies, and an extensible architecture that allows you to add your own replication features if you don’t find them there already. I’ll describe why MySQL needs more replication, show the product architecture, describe some of the problems it solves, and demonstrate the replicator in action. Finally, we’ll play “stump the chump” where you can present your replication problem and I’ll describe how our replicator handles it. Whether you have a tricky replication problem or are just curious about the subject, please join us. This is a very technical presentation that should be of interest to all true MySQL fans.

More information on Tungsten Replicator: http://www.continuent.com/community/tungsten-replicator

Peter Zaitsev will be giving his excellent presentation “Scaling MySQL powered Web Sites by Sharding and Replication” to the upcoming SF MySQL 6:00pm on November 3rd @ the CBS Interactive (formerly CNET Networks) building in SOMA San Francisco (235 2nd St).

RSVP here: http://mysql.meetup.com/30/calendar/8912109/

Description from the 2008 Velocity conference:
When your application grows beyond capacity of a single MySQL server there are few ways to scale, with most typical being Replication and Sharding.

In this presentation you will learn how, depending on your application performance scalability and high availability goals, you can choose what proper scaling strategy will be good for you.

We also will speak about scalability limitations of replication of Replication and Sharding, implementation complexities, and aspects of operation.

Peter Zaitsev’s bio:
MySQL Performance Blog

Peter helped many companies ranging from one man startups to Fortune 500 companies with their Performance and Scaling problems on various stages starting from architecture design and down to queries and schema optimization.

Over cause of the years Peter has spoken on many International conferences focused on MySQL, Open Source, Databases or High Performance Web applications.

Before co-founding Percona Peter worked for MySQL AB as head of High Performance Group being involved in Support, Consulting, Development and working with vendors helping to optimize their software or hardware to perform well with MySQL.

Peter also was involved in number of Web Startups in the roles ranging from CTO to Consultant and has a lot of experience in MySQL and Web Applications Operations, Deployment, Quality Assurance and Development.
Please join us!

Map to CNET Networks, Inc.
When you arrive, please look for the PHP/MySQL Meetup sign pointing you to the conference room.

Look forward to seeing you there!

MySQL Proxy has a c-based event driven core that has command line support, event handling and a Lua-based scripting layer. The proxy is loading into MySQL via the plugin interface. With the community edition, you are responsible for writing your own scripts. But with MySQL Enterprise, it will come with several pre-built scripts that add some enhanced functionality.

The most basic use of MySQL Proxy is to rewrite or alter your queries. If you have a costly query that is killing your db, but you’re not sure where it is coming from, you can write a Lua script to either reject the query or to alter it and make it more manageable. It stops the dba from having to wait for a developer to update code and/or deploy the fix.

Another trick you can do with MySQL Proxy is to create new SQL commands. While this is probably not going to be a great idea, you can see a few interesting examples at http://thenoyes.com/littlenoise/?p=63. It is a good way to expand on functionality at the lowest level. Or to lose/forget where you put it when you switch to another db.

One of the features MySQL was thinking of when they built MySQL Proxy is the ability to manage your queries at the database level. Imagine tracking query quotas, by ip, user, db, etc., then rejecting queries automatically if over quota. Or perhaps to simulate a db outage or transaction failure when testing code to ensure properly fail-over or error handling.

One of the default Lua scripts that comes with MySQL Proxy is a basic load balancer for the db. It rotates the connections round-robin betweeen the MySQL servers. Since it is in the database layer, it can be aware of slag lag and adjust the rotation accordingly. Since it is a Lua script, you can customize it to work how you want it to.

In MySQL Enterprise, this script is on steroids and does so much more. The load balancer automatically checks the slave state, including lag, # of connections, last error, etc., and kicks the server from rotation if it finds something “wrong” with the slave. This functionality will be integrated with MySQL Enterprise Monitor so that you will have an easy view into how the load balancer is working.

Also for Enterprise, the MySQL team is working on an automated sharding algorithm to go with MySQL 5.1′s table partitioning. I’m not sure how easy that will be to implement, but if they are able to pull it off, I’ll be mighty happy. At least for basic partitioning or sharding, like with a user table, you could do it through MySQL Proxy without having to worry about missing an update someplace in your code. If you combine this with logging, you can update your code at your leisure and easily find the queries you missed without having to worry about breaking the site.

Other planned improvements include being able to run parallel queries, two-phased commits and syncronyous replication. Since the proxy is not the database engine, it can use multiple cores on a server and assign out queries and other processes in ways that the db engine you are using cannot.

This is still very new technology/product for MySQL, but I’m looking forward to what comes out of it. There is a lot of possibility in it, as well as the potential for bloat. Personally, I think it will end up back as MySQL core so that the engines can take advantage of some of the Proxy abilities.

Maria was designed to be a crash-safe replacement for MyISAM. Maria supports the same row formats and features as MyISAM. This means it can be fully compatible with MyISAM by simply running it in non-transactional control. It will be ACID compliant and have multi-version concurrency control (MVCC).

Version 1.0 of Maria was released in January as a branch of MySQL. It will continue to be a branch until version 2.0 where it will hopefully be part of MySQL 6.  Customized documentation is also available on http://dev.mysql.com/doc/refman/5.1-maria/en/se-maria.html.

Version 1.5 of Maria should be released any week now. Monty had originally hoped to announce it this week at the conference, but it is not quite done yet. He says that it should be out sometime between Late April to mid-May. Starting with version 1.5, Maria will support concurrent INSERT and SELECT operation. This will give Maria a nice feature merge between InnoDB and MyISAM.

Starting with MySQL 6 in Q4 of 2008, version 2 of Maria will be fully transactional and ACID compliant. Having the crash recovery will be a great thing to have with the read speed of MyISAM. Version 3 will be optimized for higher concurrency.

With the expected small footprint and the efficient data storage model, Maria will be a good engine of choice for desktop applications or embedding in applications.

The way in which Maria is crash-safe is kind of interesting. It stores everything except table definitions into log files. While these log files can be quite large, starting size is 1GB, it creates a few cool features. With MyISAM tables, you can apply a lock to the db (or shutdown the server) and then copy the data to a new server. With Maria, you can simply copy the data and the logs to the new server. Like with MyISAM, the new server will have corrupted tables. Unlike MyISAM, Maria will then run through the logs and recover the db to where it should be. Or, if you don’t like that method, you can take last night’s backup to the new server, copy just today’s logs from the donor server and start up MySQL. Maria will take the data from the backup and then automatically apply the logs until the new server is up-to-date.

There is quite a list of features that are either already in Maria or will be by the time version 3 comes out.

• MVCC & ACID
• Equal lookup speed for all indexes
• Allows easy copying of tables and/or Maria logs
• Has instant COUNT(*) queries
• Better BLOG handling than MyISAM
• Versioned
• Crash safe
• Data stored in 8k pages for less locking and improved query performance

By using 8k pages Maria will be faster on static size indexes than MyISAM, but slower on variable length keys. Updates that are planned for version 2 or 3 will make variable length indexes just as fast.

There are a few disadvantages of Maria over MyISAm.

• No INSERT DELAYED (may or may not eventually be added)
• Fulltext indexes are not yet crash-safe (planned to be fixed in version 3)
• No external locking
• Like MyISAM it will not support RAID
• When using checksums, the data size will be slightly larger for indexes
• Storage of very small rows (<25 bytes) are not efficient with the new page format

All in all, I’m pretty excited about where they are going with Maria. While some of the databases I use are write heavy the vaste majority are primarily reads; there are only so many news stories a day. I looks like Maria will give us the stability and locking abilities of InnoDB while keeping the speed of MyISAM reads.

The new table partitioning system has several benefits for your large datasets where you are having performance issues due to the volume of data. It can reduce the seek & scan set sizes and reduce the INSERT/UPDATE transaction durations. If you are already performing UNION queries or secondary queries to find data in an archive table, this functionality may be for you.

There are four (point 5) partitioning styles available for your use:

1. Range
   • Very useful for situations where the data is accessed by date
   • Usually used when a limited number of primary partitions are needed
   • It supports sub-partitions (the .5 partition type)
2. List
   • Good for when you are grouping data in partitions out of order (e.g., SELECT 1,5,7 in partition X)
   • As the name implies, you list which data goes into which partition
   • Higher maintenance costs in many situations
   • Supports sub-partitions
3. Hash
   • Usually has a lower maintenance cost
   • Partitions the data based on the hash that you tell it
   • Good for when you have non-intelligent keys
   • Works with both a limited and a large number of Partitions
4. Key
   • Similar to Hash partitioning, but based on one or more of the fields in the primary key instead of a user-defined hash
   • If there is no primary key, then the first unique index is used
   • Does not require the key to be integer based
5. Subpartitions
   • As the name implies, a partition within a partition
   • You can only create subpartitions for Range & List
   • If needed, subpartitions can be spread across volumes

The alter table statements are fairly simplistic; “ALTER TABLE my_table PARTITION BY KEY() PARTITIONS 50″ will create 50 partitions for that table.  If you need to add more partitions, you can just run “ALTER TABLE my_table ADD PARTITION PARTITIONS 25″ and your table will now have 75 partitions.  Of course you can shrink the size of the partitions as well.  One of the nice features of the partition modification is that the data is re-allocated as appropriate.

All in all, this looks like a much better solution than the standard 10/90 tables.

The home for the MySQL Sandbox project is hosted on SourceForge.net. It is not an officially supported product which is why Giuseppe keeps it on SourceForge.

The Sandbox was designed so that you don’t have to go through the hassle, and potentially error prone, steps of setting up multiple sandbox databases. It’s very common for these manual installations to accidentally install in the same data directory (can corrupt your data) or install with the same port/socket (MySQL won’t start).

Details of installing and using after the break…

The simple installation steps for MySQL Sandbox automatically creates a new instance of MySQL with a sandboxed data directory, port, and socket. In order to use Sandbox’s express_install script you need to download and un-tar a copy of MySQL for each version you wish to install. The script by default uses $HOME/opt/mysql as its base directory so be sure to extract MySQL there. You can change the default source directory, but by being in your home directory it helps keep your instance separate from everyone else’s.

Next download and extract the MySQL Sandbox from SourceForge and run the following:
./express_install.pl {directory in $HOME/opt/mysql}
If you prefer, you can run ./express_install.pl /path/to/mysql/directory

You can setup replication or multiple instances by ./set_replication.pl and ./set_custom_many.pl respectively. You also have a variety of install options that can be found by adding –help to the commands.

One of the nice commands you can use once the sandbox is setup and running is “clear.” This command will reset the data and instance back to default so that you can start again from scratch. Running the “use” command will get you into the CLI of the appropriate instance. One thing to keep in mind is that the commands to interact with the sandboxes must be run from the installed directory. You also have access to some of the standard MySQL command line tools by running “my {sqldump|sqlbinlog|sqladmin}” (Note the space after my).

It seems like this would be a good system to use in conjunction with DBUnit. As I talked about yesterday, DBUnit can auto-load/populate data. By putting the two together you should have a very maintainable and separate testing environment.

Brian Aker has put the PDF of the slides at http://download.tangent.org/talks/Memcached%20Study.pdf. The slides will be updated as time goes on so this link should always have the most up-to-date stuff. In case at some point he removes them, you can find them attached.

2:15pm: Talking about Grazr and how they have a daemon to do write-through cache. Now talking about processing data so that the data normally/frequently used is always in cache.

2:25pm: Memcached is supposed to be “simple” so that it can be faster. Has its own memory slab allocator, it originally tried using malloc but that was way too slow. The way it assigns blocks for memory assignment means that each data store goes into one of those “blocks.” If the # of available room in the block is full, then it drops the oldest record in that block. Written around libevent for scalable network connections. That was only connections sending data are “active” on the server.

2:27pm: The clients handle the majority of the load. It takes the cache key and hashes it so that it knows which server to send to the data to. Server doesn’t do the serialization either.

2:30pm: Server is not redundant and does not handle failover. But some clients like with PHP’s PECL, the client can implement this type of functionality.

2:33pm: Available commands are pretty much only set/get/replace/add, append/prepend, increment/decrement, cas and stats. While they say append/prepend are easy to be abused it is still an easy & quick way to store the keys used for something. Unless the size of the keys that you are storing reaches close to 1MB, it’s okay. It’s a great way to know what keys you’ll need to clear when product information has been updated.

2:38pm: MySQL UDF usage for Memcached is growing. Google Summer of Code is working on making MySQL Query cache use Memcached. lighthttpd has mod_memcache for caching files from disk. Apache also has mod_memcached but is still alpha.

2:42pm: Memcached has a few limits that you should pay attention to:
The max cache key is 250 bytes
Max data size is 1MB
Maxbytes limits the item cache, not everything
Be careful because with 32bit machines, you can set too high of a maxbytes that in combination with other memcache memory elements (e.g., key storage) and it will segfault.

2:46pm: LRU – Least recently accessed items are up for eviction and can be seen. One LRU exists per “slab class.” LRU evictions don’t need to be common. That’s pretty nice because you won’t lose larger data sets because a small data element doesn’t have room in its slab (and vice versa).

2:50pm: Threads – Great for large instances (16G+) and/or large multiget requests. It scales okay now, but they are working on improving it. Also means that you may not need to run multiple instances on the same box. Only 1 thread can talk to the allocator and the hash table at one time. This is so that you don’t have race conditions.

2:55pm: Don’t run Memcached with swap enabled or at least set it really small. Can seriously slow down performance and technically is contrary to purpose of memcache. The smaller swap means that OS can still use if it really has to, but won’t let the writing to swap happen for Memcached. The memory for Memcached is permanently allocated from the OS. Shouldn’t be an issue with most modern servers. The slab class are created by chunk size. Tends to create 36-39 slab classes. It does not reassign slab classes once the daemon loads. They are working on a way to allow you to change the assignments on the fly. e.g., if you find that you are evicting a lot of data from one slab, you can give that slab more pages while taking it from another slab.

3:10pm: There is normal hashing (usually crc or some modulus operation) and consistent hashing. Each client could implement its own version, but usually use common methods so that multiple clients can use the same pool of memcache servers.

3:20pm: PECL client (and most others) has option to not “remove” the server if it is not available. It can have a “back-off” method where it won’t try to hit the server for 1 second, 5 seconds, then 15 seconds, etc. It can also failover to a different server until the original is back online.

3:25pm: You should always try to use multi-get. Memcached is optimized for handling multiple requests at once. You’ll find a big improvement if you do this. The trick is to write your code in a way that can utilize this to the fullest.

4:04pm: Back from break. We’re now going over various coding examples of how to use Memcached. I’m hoping this part will be helpful. Most coding examples of how to use Memcached are generally fairly simple. But with 90 minutes left, there aught to be some gems.

4:15pm: Going over locks. Much like my previous post regarding memcached cache locks.

4:25pm: Just re-hashing the same examples in different languages.

4:35pm: libmemcached is a C/C++ client. Has replicated ability. You can store flags associated with the data in the byte category. Most clients don’t allow you direct access to this.

4:48pm: When doing a multi-get it hashes each key to determine the servers where the data would be stored. Once it knows that, the requests are made in parallel and as single-combined request to each server needed.

4:50pm: MySQL & Memcached. The MySQL uses the UDF API and libmemcached. You have to install it by CREATE FUNCTION. Most common method is to use memc_delete to remove the memcached data when data is written to the db. An example is “select id, url, memc_set(concat(‘feeds’, md5(url), url) from feeds;” or “select memc_get(concat(‘feeds’, md5(url));” This could be helpful for when you store the entire row, and only the row, in memcache. But for the most part, I don’t see the benefit of using this. Of course, I’ve been known to be wrong before.

5:00pm: There is memcached-tool which may assist with some basic stats/display commands. It will eventually be what does the slab re-allocation. libmmcached has memslap which may be useful for performance testing. MRTG gives you decent graphs about what is happening on the server.

5:05pm: 1.2.5 release supports multi-interface support, UDP all the time, noreply, & IPV6. Noreply is kind of neat as it allows you to set keys and not have have a response sent back to the client. It makes it a fire & forget save to memcache. My experience is that most people don’t verify that the data is really set to memcache anyway.

I really wanted to do the MySQL Proxy session, but that is an all-day thing and I’d miss the Memcached session. In hindsight, I’m probably going to regret that decision, but we’ll see. I can always do Proxy next year.

10:00am: So far pretty much all refresher. Some people have asked some good questions about the relay logs and replication lag. Unfortunately, none of it is particularly helpful. I don’t understand why MySQL doesn’t enable a way to do master-master replication reliably outside of NDB. In my opinion, NDB, is not an appropriate solution for most websites. Having to take down the entire cluster to alter a table would not be acceptable.

10:45am: Not sure if it is my laptop or the wireless network, but my connection/authorization never seems to carry over from when I log in/out of the laptop. I have to disconnect from the wireless auto-reconnect and connect again manually in order to get on the Internet. I wonder if anyone else is having this problem?

10:45am: One of the recommend ways for doing HA master/master is to use a shared disk array. The section would be to put a virtual IP in front of a heartbeat monitor in front of two MySQL servers configured the same. If they shared the same disk array, then the bin-log will be the same for both servers. The heartbeat monitor then sends the traffic to mas1 until it finds that it is no longer online. Then the traffic goes to mas2 automatically. When mas1 comes back online, the heartbeat monitor sends the traffic back automatically. Since mas2 is no longer reading/writing to the db tables, there shouldn’t be any corruption. Behind the mas1/mas2 is another virtual IP tied to the same heartbeat monitor. That way the slaves will also stay online. This sounds like it will handle many of the scenarios where you need HA master/master. The only thing I still see missing is the ability to alter tables without bringing down the tier like you can with m/cluster. [ m/cluster is a product by Continuent for HA synchronous replication. Of course, that product is EOL and their new product doesn't allow this either. ]

11:00am: Federated databases are good for joining two sets of data to another db server. I guess I always misunderstood how that works. This seems like it’d be a good way to hook up user information with community (forums) data. I wonder what the performance costs are?

11:15am: Row-based replication in 5.1 paves the way for future replication enhancements such as conflict detection & conflict resolution. As well as multi-channel replication and horizontal partioning. Multi-channel replication would be really cool. It is such a pain when replication lags and this could go a long way to help fight that. You cannot do master filtering for individual tables with row-based. I wonder why that is?

12:00pm: I don’t know why MySQL keeps pushing NDB as the end-all solution to replication problems. Yes it can deal with a lot of the replication issues, but it has several other issues that make it a bad (or at least not optimal) solution.

12:05pm: eek, running out of battery power. I have 18 minutes left… Should be able to make it until lunch. Maybe…

12:20: That’s about it. Q&A now, but I’m headed out to find a lunch table near power.

According to Kaj Arno it will be a good thing due to Sun’s already strong presence in the open source world citing references to Java and Open Office.  I agree that Sun’s open source initiatives are a good thing (it could have been Oracle /shudder).  However, I don’t like Java and I don’t like Open Office.  I also think the audience that uses Java and MySQL might be somewhat different from the larger LAMP sites like Facebook and Yahoo!.

I suppose only time will tell.  Its probably just the db geek in me, but I’ll be watching this closely over the coming months.