Using iconv against Windows Notepad

Posted on Tuesday March 6, 2007 by Julian I. Kamil

My problems with Windows-created text files did not end with this nasty episode. I am still in the process of building the contents of a web application — writing a Ruby script that reads metadata files in a directory structure to feed the backend database with initial, static contents. The metadata are provided as YAML files, created by another team member using Windows Notepad. Simple, enough, don’t you think — no chance for any problems, right?

Wrong.

Murphy could have had Windows in mind when he came up with his law, because even this seemingly simple task of feeding Notepad-created plain text file to my Ruby script had failed miserably. This time, though, I immediately suspected incorrect character encoding as the source of the issue, and I was right.

I won’t bore you with the details of the investigation, but the bottom line is, when you save a text file in Notepad, be sure to have the right character encoding selected. If not, you could end up with a text file with characters encoded in Unicode when what you really want is just plain ASCII. Which was what happened to me, and a gross violation of the principle of least surprise.

Luckily, I do my Ruby and Rails(*) work on Xubuntu Linux, a Debian derivative. And, like most Unix platforms, it’s got iconv installed and ready to use.

iconv -fUnicode -tASCII unicode.txt -o ascii.txt

Very handy. The above command tells iconv to convert and save the Unicode file into plain ASCII.

Update: Another alternative is to use recode, which will perform an in-place update of the characters in the file.

recode windows-1252.. the-file.txt

If you are still developing Ruby and Rails applications under Windows, you should consider switching to a variant of Linux, or OS X. Everything simply works — and works better! — on these platforms, and you will also get the benefit of having access to the native implementation of wonderful GNU tools like iconv.

———
(*) No, that’s not a typo. I do a lot of naked Ruby as well as Ruby on Rails development. And I love doing both!

Commenting is closed for this article.