This came up at the breakfast table: What's the deal with "ye olde" as a quaint way of saying "The old"? That wasn't a "y", it's a printer's compromise at trying to render a "thorn", an Old English letter that was used for "th" sounds. There was another, the "eth", as well. One was for voiced "th", the other for unvoiced.

As it happens, Iceland still uses thorns and eths, so they are available in Unicode, even appearing in the first code page: Thorn: Þ (Þ) thorn: þ (þ) Eth: Ð (Ð) eth: ð (ð).