Validating HTML with tidy

Michael Tougeron — Thu, 15 Jan 2009 20:41:32 +0000

If you ever have to do HTML validation or parsing in PHP the tidy extension is the way to do it! This extension lets you use the abilities of tidy in some pretty powerful ways. The extension, written by John Coggeshall, has been around for several years now. I can see how if someone just took a quick glance at it they could think it was nice, but not really something they need. How wrong they would be! If you take a few minutes and look under the hood, tidy is an extremely powerful tool. Not only can it format html to standards (what most people use it for), it can also be a powerful parser and validation tool.

When I’m dealing with user inputted data where I want to allow HTML I have two concerns. First, I don’t want to allow XSS (some xml parsers think

” closes the

tag). Second, the user frequently enters invalid html (e.g., doesn’t close the tag). Fortunately tidy can easily deal with both. The second issue is the easiest to solve by running tidy->cleanRepair() on the html. The first is taken care of by looping through the tidy nodes and rebuilding the html using a whitelist. More about how to do this after the break.

To start, you’ll need to setup your tidy options. I tend to use the following options. The word-2000 option is great because it strips out all of the annoying html, css, etc. that Microsoft Word inserts for its formatting.

$config = array(‘indent’ => false,
‘output-xhtml’ => true,
‘wrap’ => 0,
‘fix-uri’ => true,
‘word-2000′ => true,
‘show-body-only’ => true,
‘drop-proprietary-attributes’ => true,
‘ncr’ => false,
‘drop-empty-paras’ => false,
‘hide-endtags’ => true,
‘lower-literals’ => true,
‘markup’ => true,
‘quote-ampersand’ => true,
‘force-output’ => true);

Add/Remove these options as it works best for you. I’ve found that these work best for the cleaning & repairing of accidentally invalid HTML. Read the tidy docs for what each of these mean and how they affect what tidy does. Please note that in the above options I include “force-output” so that I can display back to the user what the results of trying to fix the HTML is so that they can fix their input.

If you’re parsing user generated content, usually they are just entering the tags and HTML for the specific content they want posted. This means you’ll need to wrap that input in the , and tags before processing it through tidy.

// Let’s initialize tidy with our HTML and config options.
$tidy = tidy_parse_string($html, $config);
// Not let’s clean & repair to try and "fix" user errors
$tidy->cleanRepair();
// Start tracking any errors at this point so we can give good feedback to the user
if (tidy_error_count($tidy) && $tidy->errorBuffer ) {
$errors = explode("\n", $tidy->errorBuffer);
foreach ( $errors as $key => $error ) {
$errors[$key] = htmlentities($error);
}
}
// Once tidy has cleaned & repaired the initial user input
// we need to loop through the object and validate each
// HTML tag block. We will only need to do that though if
// the $body actually has child tags
$body = $tidy->body();
if ( $body->child ) {
// we need to validate each HTML tag in the
// validateTidyNode() will be called recursively for
// any sub-tag blocks
foreach ( $body->child as $key => $child ) {
$error = validateTidyNode($child, $new_html);
$body->child[$key] = $child;
// And of course track the errors as they
// appear to give good feedback to the user
if ( $error ) {
$errors = array_merge((array)$errors, (array)$error);
}
}
}
return array(‘html’ => $new_html, ‘errors’ => $errors);
function validateTidyNode(&$tidy_node, &$html, $allowed_tags) {
$bad_tag = false;
// If the node is text, just add the text to the HTML
if ( $tidy_node->type == TIDY_NODETYPE_TEXT ) {
$html .= $tidy_node->value;
return;
}
// If the node does not exist in your array of acceptable HTML tags
// then track and error message and set the $bad_tag flag.
elseif ( !array_key_exists(strtolower($tidy_node->name), $allowed_tags) ) {
$errors[] = ‘Tag: ‘ . $tidy_node->name . ‘ is not allowed and has been removed.’;
$bad_tag = true;
}
if ( $tidy_node->child ) {
$html2 = ”;
foreach ( $tidy_node->child as $key => $child ) {
$error = validateTidyNode($child, $html2);
$tidy_node->child[$key] = $child;
if ( $error ) {
$errors = array_merge((array)$errors, (array)$error);
}
}
}
if ( $bad_tag ) {
// I prefer to "display" the bad HTML tags so that the
// user can see that it was not accepted. But you
// may prefer to just strip the tag.
// $html .= ‘ ‘;
$html .= ‘<’ . $tidy_node->name;
}
else {
$html .= ‘<’ . $tidy_node->name;
if ( $tidy_node->attribute ) {
unset($found_attribs);
foreach ( $tidy_node->attribute as $attrib_name => $attrib_value ) {
// verify that the tag is allowed to have the specified attribute.
if ( !array_key_exists(strtolower($attrib_name), $allowed_tags[$tidy_node->name][‘attribs’]) ) {
$errors[] = ‘Tag ‘ . $tidy_node->name . ‘ is not allowed to have the attribute ‘ . $attrib_name . ‘ and has been removed.’;
unset($tidy_node->attribute[$attrib_name]);
}
else {
// validate the attribute’s value. We don’t want invalid values.
$res = validateHTMLTagAttribute($tidy_node->name, $attrib_name, $attrib_value, $allowed_tags[$tidy_node->name]);
if ( !$res[‘remove’] ) {
$html .= ‘ ‘ . $res[‘attrib_name’] . ‘="’ . $res[‘attrib_value’] . ‘"’;
}
if ( $res[‘errors’] ) {
$errors[] = $res[‘errors’];
}
}
if ( trim($attrib_name) ) {
$found_attribs[] = $attrib_name;
}
}
}
// check to make sure that required attributes are set. e.g. needs to have the src attribute
if ( $allowed_tags[$tidy_node->name][‘attribs’] ) {
foreach ( $allowed_tags[$tidy_node->name][‘attribs’] as $attrib_name => $attrib_settings ) {
if ( $attrib_settings[‘required’] && (!$found_attribs || !in_array($attrib_name, $found_attribs)) ) {
$errors[] = ‘Tag ‘ . $tidy_node->name . ‘ is required to have the attribute ‘ . $attrib_name . ‘.’;
}
}
}
// some tags require at least one of a set of attributes. e.g. needs to have either the href or name attribute.

if ( $allowed_tags[$tidy_node->name][‘required_attribs’] ) {

$attrib_found = false;

if ( $found_attribs ) {

foreach ( $allowed_tags[$tidy_node->name][‘required_attribs’] as $attrib_name ) {

if ( in_array($attrib_name, $found_attribs) ) {
$attrib_found = true;
}
}
}
if ( !$attrib_found ) {
$errors[] = ‘Tag ‘ . $tidy_node->name . ‘ is required to have one of the following attributes: ‘ . implode(‘, ‘, $allowed_tags[$tidy_node->name][‘required_attribs’]) . ‘.’;
}
}
}
if ( isset($allowed_tags[$tidy_node->name][‘settings’]) && !$allowed_tags[$tidy_node->name][‘settings’][‘require_close’] ) {
$html .= ‘ /’;
}
if ( $bad_tag ) {
// $html .= ‘ ‘;
$html .= ‘>’;
}
else {
$html .= ‘>’;
}
$html .= $html2;
if ( $bad_tag || $allowed_tags[$tidy_node->name][‘settings’][‘require_close’] ) {
if ( $bad_tag ) {
$html .= ‘</’ . $tidy_node->name . ‘>’;
}
else {
$html .= ‘ . $tidy_node->name . ‘>’;
}
}
return $errors;
}
function validateHTMLTagAttribute( $tag_name, $attrib_name, $attrib_value, $tag_data ) {
// set to the unmodified version first. if modified, it will change at the end of the method.
$res[‘attrib_name’] = $attrib_name;
$res[‘attrib_value’] = $attrib_value;
if ( !$tag_data) {
$res[errors] = "$tag_name could not be found. Invalid tag/attribute.";
return $res;
}
if ( $tag_data[‘attribs’][‘unlimited_attribs’] ) {
$res[‘valid’] = true;
return $res;
}
$attrib = $tag_data[‘attribs’][$attrib_name];
switch ( $attrib[‘type’] ) {
// no restrictions on the value of the attributes (not normally recommended)
case ‘unrestricted’:
break;
// the attribute can only have one of the defined values
case ‘fixed’:
if ( !in_array(strtolower($attrib_value), $attrib[‘values’]) ) {
$res[‘errors’] = "$attrib_name is not set to an accepted value for $tag_name tag.";
return $res;
}
break;
// some basic numeric checks
case ‘numeric’:
case ‘px’:
case ‘percent’:
$attrib_value = intval($attrib_value);
if ( $attrib_value > $attrib[‘max’] ) {
$res[‘errors’] = "$attrib_name is greater than the max value (" . $attrib[‘max’] . ") allowed for $tag_name tag.";
return $res;
}
elseif ( $attrib_value < $attrib[‘min’] ) {
$res[‘errors’] = "$attrib_name is less than the max value (" . $attrib[‘min’] . ") allowed for $tag_name tag.";
return $res;
}
if ( $attrib[‘type’] == ‘percent’ ) {
$attrib_value = $attrib_value . "%";
}
elseif ( $attrib[‘type’] == ‘px’ ) {
$attrib_value = $attrib_value . "px";
}
break;
case ‘hex’:
// 6 characters of A-F and 0-9
// optionally allow a # in front for a total
// of 7 characters
break;
case ‘url’:
// validate according to your valid URL criteria
// e.g., only allow for current domain or if the
// tag is "img" make sure the extension is .jpg or .gif
break;
case ‘string’:
// validate for a string. I usually consider _-. and similar
// characters acceptable for a "string" even though they are
// not truly a string
break;
default:
$res[‘errors’] = "ERROR! Do not know how to process the tag $tag_name!";
break;
}
$res[‘attrib_name’] = $attrib_name;
$res[‘attrib_value’] = $attrib_value;
$res[‘valid’] = true;
return $res;
}

So how you setup the array for tag validation is pretty much up to you. The format I used looks a lot like this:

$tags = array(
‘a’ => array( ‘settings’ => array(‘require_close’ => true),
‘attribs’ => array( ‘href’ => array(‘type’ => ‘url’),
‘name’ => array(‘type’ => ‘string’),
‘id’ => array( ‘type’ => ‘string’,
‘auth_level’ => AUTH_LEVEL_TRUSTED_USER
),
‘class’ => array( ‘type’ => ‘unrestricted’,
‘auth_level’ => AUTH_LEVEL_TRUSTED_USER
),
‘onclick’ => array( ‘type’ => ‘unrestricted’,
‘auth_level’ => AUTH_LEVEL_TRUSTED_USER
),
‘onmouseover’ => array( ‘type’ => ‘unrestricted’,
‘auth_level’ => AUTH_LEVEL_TRUSTED_USER
)
),
‘required_attribs’ => array( ‘href’, ‘name’ )
),
‘b’ => array(‘settings’ => array( ‘require_close’ => true)),
‘img’ => array( ‘settings’ => array( ‘require_close’ => false),
‘attribs’ => array( ‘src’ => array( ‘type’ => ‘url’,
‘required’ => true
),
‘alt’ => array(‘type’ => ‘string’),
‘border’ => array( ‘type’ => ‘numeric’,
‘min’ => 0,
‘max’ => 5
),
‘align’ => array( ‘type’ => ‘fixed’,
‘values’ => array( ‘center’,
‘left’,
‘right’
)
),
‘height’ => array( ‘type’ => ‘numeric’,
‘min’ => 1,
‘max’ => 2048
),
‘width’ => array( ‘type’ => ‘numeric’,
‘min’ => 1,
‘max’ => 2048
),
‘title’ => array(‘type’=>‘string’),
‘hspace’ => array( ‘type’ => ‘numeric’,
‘min’ => 0,
‘max’ => 10
),
‘vspace’ => array( ‘type’ => ‘numeric’,
‘min’ => 0,
‘max’ => 10
),
‘id’ => array( ‘type’ => ‘string’,
‘auth_level’ => AUTH_LEVEL_TRUSTED_USER
),
‘class’ => array( ‘type’ => ‘unrestricted’,
‘auth_level’ => AUTH_LEVEL_TRUSTED_USER
),
‘onclick’ => array( ‘type’ => ‘unrestricted’,
‘auth_level’ => AUTH_LEVEL_TRUSTED_USER
),
‘onmouseover’ => array( ‘type’ => ‘unrestricted’,
‘auth_level’ => AUTH_LEVEL_TRUSTED_USER
)
)
),
);

The auth_level is a filter that I have in place to only allow users with a certain amount of access, such as internal staff, to use that attribute. As you probably already know, attributes like onmouseover are an easy gateway to XSS exploits. As you add new elements to the tag rule definition you just need to update the validate function in order to validate against it.

You may have noticed that I’m using the deprecated HTML attributes instead of putting all of this in a “style” attribute. I’ve found that most users don’t know CSS styles and are familiar with the old-style HTML attributes. If you force your users to use styles you’ll have whole other validation routine to ensure exploits are not in the CSS.

Note: Please don’t just copy/paste this code into your code and use it. I’m quite positive there are bugs and typos all throughout. Oh, and no, this is not the code used by the sites I work on. It is similar but definitely not the same.

Grep My Mind » Security

Validating HTML with tidy