What's the difference between speed and velocity?

The short answer is that velocity is the speed with a direction, while speed does not have a direction.
Speed is a scalar quantity -- it is the magnitude of the velocity. Speed is measured in units of distance divided by time, e.g. miles per hour, feet per second, meters per second, etc....
Velocity is a vector quantity -- when giving the velocity we must specify the magnitude (the speed) and the direction of travel....

Just as distance and displacement have distinctly different meanings (despite their similarities), so do speed and velocity. Speed is a scalar quantity that refers to "how fast an object is moving." Speed can be thought of as the rate at which an object covers distance. A fast-moving object has a high speed and covers a relatively large distance in a short amount of time. Contrast this to a slow-moving object that has a low speed; it covers a relatively small amount of distance in the same amount of time. An object with no movement at all has a zero speed.

Velocity as a Vector Quantity

Velocity is a vector quantity that refers to "the rate at which an object changes its position." Imagine a person moving rapidly - one step forward and one step back - always returning to the original starting position. While this might result in a frenzy of activity, it would result in a zero velocity. Because the person always returns to the original position, the motion would never result in a change in position. Since velocity is defined as the rate at which the position changes, this motion results in zero velocity. If a person in motion wishes to maximize their velocity, then that person must make every effort to maximize the amount that they are displaced from their original position. Every step must go into moving that person further from where he or she started. For certain, the person should never change directions and begin to return to the starting position.
Velocity is a vector quantity. As such, velocity is direction aware. When evaluating the velocity of an object, one must keep track of direction. It would not be enough to say that an object has a velocity of 55 mi/hr. One must include direction information in order to fully describe the velocity of the object. For instance, you must describe an object's velocity as being 55 mi/hr, east. This is one of the essential differences between speed and velocity. Speed is a scalar quantity and does not keep track of direction; velocity is a vector quantity and is direction aware.