mysql/mariadb supports utf-8 encoding within DB tables, but it doesn't support UTF-8 in /etc/my.cnf???

A few weeks ago, I installed RHEL 6.6 and MariaDB 10.X at a client site but had problems running a DB install script. The message I got was

found option without preceding group in config file /etc/my.cnf at line 1

A google search for this snippet turned up the following StackOverflow thread:

http://stackoverflow.com/questions/8020297/mysql-my-cnf-file-found-option-without-preceding-group

Apparently, the /etc/my.cnf config file for mysql/mariadb only supports ASCII! This was quite a shock, because I know UTF-8 support is now built-into both databases.

I verified that /etc/my.cnf was indeed encoded as UTF-8 text using the file utility:

$ file /etc/my.cnf
/etc/my.cnf: utf-8

To convert text within a file from one encoding to another, use the iconv utility:

iconv -c -f utf8 -t ascii /etc/my.cnf

Explanation of the option flags (from man iconv):
-c   silently discard chars that cannot be converted instead of terminating when encountering such chars

-f   from-encoding

-t   to-encoding

In 2015, virtually all POSIX programs get along just fine with UTF8, so why does mysql/mariadb have a problem with UTF8 in my.cnf? This bit of techno-trivia has now been added to my growing body Linux sysadmin lore, but this should issue should be fixed, in my opinion.

Webscraping pages containing non-UTF-8 CJK text with BeautifulSoup 4 뷰티플수프로 텍스트 추출하기

In the past, I have written some posts (JS: 오른쪽 마우스 클릭 차단 그만! and JS: 오른쪽 마우스 클릭 차단 Pt II) in Korean about how to apply Greasemonkey user scripts to non-programmatically copy text from web pages which have disabled right-click and copy-paste.

The blogging platforms popular in Korea (Naver, Daum/Tistory, etc) are referred to as "카페" (cafés) and generally block right-click (oncontextmenu) by default. The content I am interested in copying from some of these blog cafés includes public-domain classical Korean poems for which the copyright obviously doesn't reside with the blog itself. Such content shouldn't be locked up behind Javascript that disables basic browsing features.

The problem with Greasemonkey user scripts like Anti-Disabler, however, is that they don't work on all sites and aren't updated often enough to deal with changes in the anti-copying Javascript plugins from Daum and Naver.

Here's where BeautifulSoup comes in handy. Using bs4 (BeautifulSoup 4.2.0) for Python 3 (which uses UTF-8 by default, great for CJK), I scraped an article from a Korean news site as well as from a locked-down blog. Here's some sample code:

Korean news sites use tons of banner ads reminiscent of densely packed neon lights from entertainment districts in Gangnam, Hong Kong, or Tokyo along with funky CSS layout that sometimes make it hard to copy-and-paste an entire article. With Beautiful Soup, we can avoid all the bling and just get pure text.

Here's the .get_text() output of the whole article from Chosun.com:


Beautiful Soup also works great on right-click disabled web pages. Here's a snippet of text from an article about SEO for the Korean search engine Naver:


Note: Beware of possible encoding problems when you save .html files locally and try to parse them with BeautifulSoup using the open() method. Many webpages written in Chinese Japanese Korean (CJK) are still not encoded in UTF-8, instead using older formats such as SHIFT JIS, GBK, EUC-KR, and various Code Pages for Asian languages. These encodings are properly detected and decoded by BeautifulSoup, but the problem occurs when your system locale differs from the encoding of the .html file you are trying to save.

For example, my desktop Linux system uses en_US.UTF-8 for LANG and LC_... settings. Therefore when I save a text file with a non-UTF-8 encoding like EUC-KR, it is automatically saved as en_US.UTF-8, the current locale! The problem is that the EUC-KR encodings are invalid as UTF-8, so when you try to parse the .html file with BeautifulSoup, you will get the following error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x0x... in position 123: invalid start byte

Since the file has been saved as UTF-8, BeautifulSoup expects to find that encoding, but chokes when it finds EUC-KR instead. When opening a URL, by contrast, BeautifulSoup doesn't run into this problem of inconsistent encodings.

I have yet to succeed at using BeautifulSoup on a EUC-KR encoded webpage saved locally with that encoding. In Emacs, I specify the encoding for the file to be saved with C-x C-m f RET euc-kr RET but when I run file --mime localFile.html the console tells me the file is encoded as Latin-1 iso-8859-1!

How to fix broken CJK filenames extracted from zip archive created in MS Windows environment

Most computers run some form of MS Windows, which doesn't natively use UTF-8 for character encoding. This can cause problems for Linux users who have to work with filenames using East Asian CJK (Chinese Japanese Korean) characters from a Windows environment. For single files sent as email attachments through Gmail, Google is smart enough to detect what code page the filename is encoded in and convert it to UTF-8 when the attachment is downloaded to a POSIX environment.

For archive files like .zip, however, compressed files named using CJK characters in a MS Windows environment will appear as gibberish in a UTF-8 locale.

Rather than booting up a Windows VM just to extract files from an archive, a faster method is to extract the compressed files while maintaining their original filename character encoding.

The following example will use this .zip file that was created on a Korean version of MS Windows. Korean language characters on Windows are encoded using Code Page 949, which is compatible with EUC-KR, the most-widely used character encoding in Korea.

First I will extract the file using 7z from the CLI, but create a modified environment with a different language encoding by using env and the LANG=... flag. This method was first described by developer Allen Choong in this post from 2013 in which he details converting filenames encoded in MS Windows GBK (Code Page 936) Simplified Chinese to UTF-8 after extraction from an archive file.

[archjun@arch Downloads]$ env LANG=C 7z x 편혜영.zip

7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=C,Utf16=off,HugeFiles=on,2 CPUs)

Processing archive: 편혜영.zip

Extracting  Korean Writers(2009)/������(����).doc.docx
Extracting  Korean Writers(2009)/������.hwp

Everything is Ok

Files: 2
Size:       33991
Compressed: 25578

You can see that the filenames extracted from the archive are mangled, as they have a non-UTF8 character encoding.

Note that the LANG variable can also be set to euc-kr or cp949 instead of C which will also maintain the original filename character encoding for archive files created in Korean Windows.

Next we need to convert the gibberish filenames from EUC-KR/CP949 to UTF-8 using convmv, which according to the description on its man page:

converts filenames from one encoding to another 

[archjun@arch Downloads]$ convmv -f cp949 -t utf8 -r --notest ~/Downloads/"Korean Writers(2009)"/

mv "/home/archjun/Downloads/Korean Writers(2009)/������(����).doc.docx" "/home/archjun/Downloads/Korean Writers(2009)/편혜영(영문).doc.docx"

mv "/home/archjun/Downloads/Korean Writers(2009)/������.hwp" "/home/archjun/Downloads/Korean Writers(2009)/편혜영.hwp"

Ready!

In the -f (from language) flag, you can also use euc-kr and the filename conversion will work just fine. The -r flag tells convmv to convert all filenames recursively (all files in the directory or sub-directories).

The --notest flag must be added for convmv to actually overwrite the existing filenames. As you can see above, the � gibberish characters have been converted to readable Korean.

In Allen's original post referred to above, he makes the important point that if you just naively extract an archive that contains filenames encoded in non-UTF8 characters onto a system with a UTF-8 locale, the gibberish filenames will automatically be encoded as UTF-8 but still be unreadable. If this happens, you will not be able to convert the mangled filenames to UTF-8 because they are in UTF-8 already!

For example,

[archjun@arch Downloads]$ 7z x 편혜영.zip

7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18

p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,2 CPUs)

Processing archive: 편혜영.zip

Extracting  Korean Writers(2009)/ÆíÇý¿µ(¿µ¹®).doc.docx

Extracting  Korean Writers(2009)/ÆíÇý¿µ.hwp

Everything is Ok

Files: 2

Size:       33991

Compressed: 25578

[archjun@arch Downloads]$ convmv -f cp949 -t utf8 -r --notest ~/Downloads/"Korean Writers(2009)"/

Skipping, already UTF-8: /home/archjun/Downloads/Korean Writers(2009)/ÆíÇý¿µ(¿µ¹®).doc.docx

Skipping, already UTF-8: /home/archjun/Downloads/Korean Writers(2009)/ÆíÇý¿µ.hwp

Ready!convmv -f cp949 -t utf8 -r --notest ~/Downloads/"Korean Writers(2009)"/

In the case above, we didn't specify a character encoding for the extracted filenames, so 7z defaults to the character encoding in our locale, which is en_US.UTF-8

Because of this, when we try to use convmv to convert from a MS Windows character encoding to UTF-8, convmv tells us that the filenames are already in UTF-8 and therefore cannot be converted!

The same holds true for other archive extractors like unzip, file-roller, etc. So don't forget to preface the archive extraction command with env to create a modified environment and then set LANG to the proper encoding (whether it is euc-jp, euc-kr, shift_jis, gbk, etc.) so that the extracted filenames' original character encoding will be maintained, thereby enabling conversion with convmv!

Postscript 2014-12-21:

Once you have converted filenames from a Windows text encoding like euc-kr to UTF-8, you may also need to convert text within a pure text file (not a binary like .doc, .hwp, etc) created in a Windows environment into UTF-8.

The Linux command for converting text within a file to another encoding is iconv. Let's assume we have a file, someText.txt, that was created in Windows and that contains Korean characters encoded in euc-kr. To convert to UTF-8 you can invoke iconv with the following flags:

iconv -c -f euc-kr -t utf8 someText.txt > someTextUTF-8.txt

-c  Silently discard characters that cannot be converted instead of
      terminating when encountering such characters.

-f  from-encoding (input characters)

-t  to-encoding (output characters)

The invocation above reads in someText.txt in euc-kr encoding and redirects output to someTextUTF-8.txt in UTF-8 encoding.