A quick-and-dirty Python3 script to extract only English or non-English characters from a textfile

When I was a full-time interpreter/translator, once in a while clients would send me proofreading work in the form of dual-language files containing alternating lines of Korean followed by English. The problem is that translators bill their clients by counting the number of words in the source language, but this is not possible when both source and target language are mixed up in the same file!

Imagine that you have a 100-page .doc file with alternating lines of English and some foreign language. It is not feasible to manually cut-and-paste all the foreign language sentences into another file! Luckily, Python 3 exists and it is UTF-8 friendly, so we can easily manipulate English and all kinds of foreign languages within Python 3 programs.

My script is called deleteLanguage.py and it is available on github at https://github.com/gojun077/deleteLanguage.

It will take mixed text like (I didn't do this horrible translation into English, btw)

기업에서 왜 트리즈를 교육해야 하는가?
Why do the companies should educate TRIZ to their members?
오늘날 특히 기업에서의 연구개발은 문제를 해결하느냐 못하느냐의 문제가 아니다.
Today being able to solve the problems or not being isn’t a real problem in the corporation’s research and development.
얼마나 빨리 새로운 결과를 찾아내는 가에 따라 성공여부가 결정된다.
The success of them depends on how fast they can find the new solutions.
하지만 우리들은 문제를 더 빨리 혁신적으로 해결할 수 있는 방법을 공부한 적이 없다.
But we have never learned to solve problems faster and more innovative.
대부분의 많은 연구개발자들은 창의적인 문제해결이 무엇인지도 모른다.
Most researchers and engineers don’t even know what the creative method to solve the problems is.
오늘날도 많은 연구자들은 각자 기존의 경험과 지식을 바탕으로 열심히 생각하기를 한다.
Today they are thinking hard based on only their own experience and knowledge.

and parse it into separate English

Why do the companies should educate TRIZ to their members?
Today being able to solve the problems or not being a real problem in the research and development.
The success of them depends on how fast they can find the new solutions.
But we have never learned to solve problems faster and more innovative.
Most researchers and engineers even know what the creative method to solve the problems is.
Today they are thinking hard based on only their own experience and knowledge.

and non-English output:

기업에서 왜 트리즈를 교육해야 하는가?
오늘날 특히 기업에서의 연구개발은 문제를 해결하느냐 못하느냐의 문제가 아니다.
얼마나 빨리 새로운 결과를 찾아내는 가에 따라 성공여부가 결정된다.
하지만 우리들은 문제를 더 빨리 혁신적으로 해결할 수 있는 방법을 공부한 적이 없다.
대부분의 많은 연구개발자들은 창의적인 문제해결이 무엇인지도 모른다.
오늘날도 많은 연구자들은 각자 기존의 경험과 지식을 바탕으로 열심히 생각하기를 한다.

The version in the initial commit has the following limitations:

The script will assume that non-ASCII characters not included in string.printable (from the Python string module) are non-English characters, so the following strings

'Awesome☻'
'...noted.†'

would not be detected as 'English' by the script.

In non-English sentences containing a the occasional English word, the script just omits these words entirely. Consider the following Korean sentence:


"철수씨는 IBM에 근무한다."

deleteLanguage.py as it is currently implemented will parse the above snippet into the following when it outputs the non-English only text file:

"철수씨는 근무한다."

The '에' character adjoining IBM is deleted along with the English word.

I haven't yet thought up a sure-fire algorithm to avoid this problem; creating prescriptive rules for dozens of one-off cases doesn't seem to be the solution, either.