Re: newsletters in text format

Bier de Stone  wrote in message
<kxASd.259063$w62...@bgtnsc05-news.ops.worldnet.att.net>:
> OK. I know I may have been caught dozing in computer lab when the 
> teacher was lecturing on file format. I take your response as a HELL NO! 
> Unicode-16 is bigger and uglier than Unicode-8. You have to admit, 
> though, the whole idea that ascii characters being a limited version of 
> the alphabet, doesn't apply as well to the font. I mean, that's where 
> you lost me.
> 
> OK, the first 128 characters of the unicode standard is the ascii 
> standard AND characters 32 thru 126 are used for this NG. I don't see 
> where you specify that characters 32 thru 126 have got to be fixed 
> width. Isn`t a character a character? Or is that what distinguishes 
> ascii from Western, Unicode-8, and Unicode-16? In other words, ascii not 
> only encompasses particular letters and symbols, it also includes the 
> fonts in which they are displayed.

I think you need some theory here. At the beginning, you have characters.
Defining exactly what is a characters is somewhat complicated (is the French
oe ligature one or two characters? and what about complex Indian
ligatures?), but it does not concern programmers: it is a linguist work.
Linguist provide a list of characters, they define them anyhow, and the task
of the programmers is to devise a way to encode a sequence of characters
into a sequence of bytes in an unambiguous way. Such a correspondance is
called an encoding.

ASCII is one of the first efforts to have a standard encoding. At this time,
programmers speak English, and the bytes are reliably 7 bits wide, so the
code chosen use one byte per character, and allow to code the characters
required for basic English.

Some time later, the standard of 8 bits bytes emerged, and
non-English-speaking had started using computer, so there was a rush for the
128 more bytes values to code various language-specific characters. Most of
these encodings have this in common: there is always a one-to-one relation
between bytes and characters.

There is more: this relation is almost always the same as ASCII for the
values between 0 and 127. This has an advantage: consider a protocol that
understands various ASCII characters as markup (the quote for a litteral
string, for example), and treates every other character with the rule "I do
not know, I let it through". If the software handling such a protocol is not
too badly written, the "I do not know, I let it through" rule will apply
also to the bytes values beyond 128. Such programs and protocols are called
8bits clean.

With one byte per characters encodings which extends ASCII, there is no need
for intermediary software to know the encodings: the input software has to
know it, the output software has to know it too, but it is enough for the
intermediary software to be 8bits clean: all non-ASCII characters will just
go through unchanged. Such encodings are called compatible with ASCII.

Well, 8 bits bytes means 256 bytes values, that is not enough for Chinese or
Japanese. Chinese and Japanese programmers invented encodings with several
bytes per character, sometimes compatible with ASCII (for exemple the ASCII
value if the character is available in ASCII, and two 128-255 bytes else).

Unicode is an effort to unify everything.

First of all, Unicode is not an encoding.

I repeat: UNICODE IS NOT AN ENCODING.

Unicode is a collection of characters. It is a big collection, and intends
to be universal, but there are no bytes values involved. The characters in
the collection are numbered, because it is very useful, but it is a
mathematical numbering, it is not theorically bounded (but alas, it is
politically bounded, see later).

As such, Unicode can be used as a bridge between encodings. For exemple,
take the Euro currency symbol. It has code 0x80 in the windows-1252
encoding, 0xA4 in the ISO-8859-15 encoding, 0xAC in the LaTeX T1 encoding,
and so on. How to handle that without having tables for every single pair of
encodings? Unicode provides a way: the only need is a table to match each
encoding with Unicode.

But there is more: along with Unicode were designed encodings to logically
code the Unicode numbers into bytes sequences.

At the time before one descovered a mine of more than sixty thousand old
Chinese ideograms, one thought that 16 bits was enough to code all, so one
used 2-bytes integers to code the Unicode characters. This is UCS-2. Well,
this is UCS-2 big endian or UCS-2 little endian. This is always a problem.

Now, anyone thinks that 2^32 will be enough for everyone, so we have UCS-4:
4-bytes integers to code the Unicode characters. And again, the endianness
problem.

Alas, a lot of programs used UCS-2, so one invented a dirty hack to extend
it: a few unaffected values were used as "surrogates", and along with a
second value allow to code up to about one million values. This is UTF-16.
UTF-16 is a verry bad standard. And as usual with bad standards, it is
heavily supported by a few big companies writing bad software. In this case:
microsoft, Sun (with java), IBM. To avoid upsetting these monsters, the
Unicode consortium has decided never to go beyond the possibilities of
UTF-16.

There is also UTF-8. It is a bit complicated, with characters using between
one and six bytes, to code values up to 2^31, but it has a lot of
advantages:

- it is compatible with ASCII, so it can be handled by 8bit-clean programs;

- the lexical order of its bytes is the same as the lexical order of the
  characters they code;

- it is corruption-proof: if some bytes are lost or wrong, it is possible to
  recover at the next uncorrupted character;

- it is quite compact for western languages, with not much more than one
  byte per character;

- it has no endianness problem.

UTF-8 is the Good encoding to use nowadays.

Now, for your macintosh gibberish: mac roman, as far as I know, is a
one-byte-per-character ASCII-compatible encoding used by apple, since they
never can do anything like others. Unicode-8 and Unicode-16 does not mean
anything at all, but I supose the first should be written "UTF-8" and the
second either "UTF-16" or "UCS-2".

Now, why those groups do not like Unicode. Well, consider the smileys. To a
certain extent, the first who spotted that a colon, a dash and a parenthesis
looked like a smiling face was a genius. But it does not require a genius
to see that the character numbered 9786 in Unicode is a smiling face; it is
even written in its name, "WHITE SMILING FACE". Somehow, using Unicode to
make so-called-ASCII-art is cheating.

As for the last question: you can use proportional-spaced fonts and do
characters-based art, but it will be completely specific to the font you
used. Change the font, the drawing will be completely wrong. And you have no
way to describe simply the spacing of a font: you have to write the whole
metrics. Except for monospaced fonts: there is nothing to say beyond that
word: monospaced. Furthermore, monospaced fonts are the historical support
for ASCII-art: people used to do ASCII-art on terminals and line printers a
long time before they were capable of displaying proportional-spaced fonts.