Safe characters for friendly url

I need to make a website that will have articles, and I would like to make friendly URLs for it, example the URL of the page with

Title: Article Test

should become: http://www.example.com/articles/article_test.

Of course I need to remove some characters from the title like ? or #, but I'm not sure which ones to remove.

Can someone tell me what characters are safe to keep?

Answers


To quote section 2.3 of RFC 3986:

"Characters that are allowed in a URI but do not have a reserved purpose are called unreserved. These include uppercase and lowercase letters, decimal digits, hyphen, period, underscore, and tilde."

ALPHA  DIGIT  "-" / "." / "_" / "~"

Note that RFC 3986 lists fewer reserved punctuation marks than the older RFC 2396.


There are two sets of characters you need to watch out for: reserved and unsafe.

The reserved characters are:

  • ampersand ("&")
  • dollar ("$")
  • plus sign ("+")
  • comma (",")
  • forward slash ("/")
  • colon (":")
  • semi-colon (";")
  • equals ("=")
  • question mark ("?")
  • 'At' symbol ("@")
  • pound ("#").

The characters generally considered unsafe are:

  • space (" ")
  • less than and greater than ("<>")
  • open and close brackets ("[]")
  • open and close braces ("{}")
  • pipe ("|")
  • backslash ("\")
  • caret ("^")
  • percent ("%")

I may have forgotten one or more, which leads to me echoing Carl V's answer. In the long run you are probably better off using a "white list" of allowed characters and then encoding the string rather than trying to stay abreast of characters that are disallowed by servers and systems.


You are best keeping only some characters (whitelist) instead of removing certain characters (blacklist).

You can technically allow any character, just as long as you properly encode it. But, to answer in the spirit of the question, you should only allow these characters:

  1. Lower case letters (convert upper case to lower)
  2. Numbers, 0 through 9
  3. A dash - or underscore _
  4. Tilde ~

Everything else has a potentially special meaning. For example, you may think you can use +, but it can be replaced with a space. & is dangerous, too, especially if using some rewrite rules.

As with the other comments, check out the standards and specifications for complete details.


Always Safe

These are safe (in theory/spec), basically anywhere except the domain name.Percent-encode anything not listed, and you're good to go.

    A-Z a-z 0-9 - . _ ~ ( ) ' ! * : @ , ;

Sometimes Safe

Only safe when used within specific URL components; use with care.

    Paths:     + & =
    Queries:   ? /
    Fragments: ? / # + & =

Never Safe

According to the URI spec (RFC 3986), all other characters must be percent-encoded. This includes:

    <space> <control-characters> <extended-ascii> <unicode>
    % < > [ ] { } | \ ^

If maximum compatibility is a concern, limit the charset to A-Z a-z 0-9 - _ . (with periods only for filename extensions).


Looking at RFC3986 - Uniform Resource Identifier (URI): Generic Syntax, your question revolves around the path component of a URI.

    foo://example.com:8042/over/there?name=ferret#nose
     \_/   \______________/\_________/ \_________/ \__/
      |           |            |            |        |
   scheme     authority       path        query   fragment
      |   _____________________|__
     / \ /                        \
     urn:example:animal:ferret:nose

Citing section 3.3, valid characters for a URI segment are of type pchar:

pchar = unreserved / pct-encoded / sub-delims / ":" / "@"

Which breaks down to:

ALPHA / DIGIT / "-" / "." / "_" / "~"
pct-encoded
"!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "="
":" / "@"

Or in other words: You may use any (non-control-) character from the ASCII table, except /, ?, #, [ and ].

This understanding is backed by RFC1738 - Uniform Resource Locators (URL).


unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~"


From the context you describe, I suspect that what you're actually trying to make is something called an 'SEO slug'. The best general known practice for those is:

  1. Convert to lower-case
  2. Convert entire sequences of characters other than a-z and 0-9 to one hyphen (-) (not underscores)
  3. Remove 'stop words' from the URL, i.e. not-meaningfully-indexable words like 'a', 'an', and 'the'; Google 'stop words' for extensive lists

So, as an example, an article titled "The Usage of !@%$* to Represent Swearing In Comics" would get a slug of "usage-represent-swearing-comics".


The format for an URI is defined in RFC 3986. See section 3.3 for details.


From an SEO perspective, hyphens are preferred over underscores. Convert to lowercase, remove all apostrophes, then replace all non-alphanumeric strings of characters with a single hyphen. Trim excess hyphens off the start and finish.


I had similar problem, I wanted to have pretty urls and reached to the conclusion that I have to allow only letters, digits, - and _ in urls. That is fine, then I wrote some nice regex and I realized that it recognizes all UTF8 chars are not letters in .NET and was screwed. This appears to be a know problem for .NET regex engine. SO I got to this solution:

private static string GetTitleForUrlDisplay(string title)
{
    if (!string.IsNullOrEmpty(title))
    {
        return Regex.Replace(Regex.Replace(title, @"[^A-Za-z0-9_-]", new MatchEvaluator(CharacterTester)).Replace(' ', '-').TrimStart('-').TrimEnd('-'), "[-]+", "-").ToLower();
    }
    return string.Empty;
}


/// <summary>
/// All characters that do not match the patter, will get to this method, i.e. useful for unicode chars, because
/// .NET impl of regext do not handle unicode chars. So we use char.IsLetterOrDigit() which works nicely and we 
/// return what we approve and return - for everything else.
/// </summary>
/// <param name="m"></param>
/// <returns></returns>
private static string CharacterTester(Match m)
{
    string x = m.ToString();
    if (x.Length > 0 && char.IsLetterOrDigit(x[0]))
    {
        return x.ToLower();
    }
    else
    {
        return "-";
    }
}

I found it very useful to encode my url to a safe one when I was returning a value through ajax/php to a url which was then read by the page again.

PHP output with url encoder for the special character &

//PHP returning the sucess info of ajax request
echo "".str_replace('&','%26',$_POST['name'])." category was changed";

//javascript sending the value to url
window.location.href='time.php?return=updated&val='+msg;

//javascript/php executing the function printing the value of the url,
//now with the text normally lost in space because of the reserved & character.

setTimeout("infoApp('updated','<?php echo $_GET['val'];?>');",360);

I think you're looking for something like "URL Encoding" - encoding a URL so that it's "safe" to use on the web:

Here's a reference for that. If you don't want any special characters, just remove any that require URL encoding:

http://www.w3schools.com/TAGS/ref_urlencode.asp


Between 3-50 characters. Can contain lowercase letters, numbers and special characters - dot(.), dash(-), underscore(_) and at the rate(@).


Need Your Help

Modify app settings used by external DLL at runtime

c# .net asp.net-mvc configuration app-config

I'm trying to build a simple client for testing an internal authentication service. The main idea is to just have a simple tool for testing to ensure that this service is callable (and successfully

Celery Tasks not showing in Django Admin

django celery django-celery

I've followed the all the docs I could find as closely as I could, even setting up an example project using Celery's official django example (celery/examples/django) but tasks still aren't showing ...