2014년 7월 15일 화요일

smtp-cli - send email from the Command Line without a mail server

smtp-cli is a Perl script that enables you to send mail from the command line without having to set up a local mail server. It can interface with webmail smtp servers like smtp.gmail.com and supports TLS.

A list of optional flags including --user --pass --from --to --subject can be seen at the bottom of the smtp-cli Perl script on Github.

Some caveats when trying to send mail through Gmail:

1. When you use the flags --user username --pass foo you cannot use your regular PW for Gmail but must generate an application-specific password instead.

If you don't specify a password beforehand with --pass, you will be interactively prompted to enter your PW as soon as smtp-cli runs

Enter password for username@smtp.gmail.com :

2. If you want to set up a cron job to run smtp-cli, make sure you specify your app-specific password after the --pass flag, otherwise cron will not be able to handle the interactive prompt.

3. To be able to send multilingual emails, be sure to pass the flag
--charset=UTF-8


Here is an example invocation of smtp-cli that will send an email to my personal gmail account:

[archjun@arch smtp-cli]$ smtp-cli --verbose --host=smtp.gmail.com --enable-auth --user gojun077 --pass 123456789abcdefg --from gojun077@gmail.com --to gojun077@gmail.com --subject "testing smtp-cli script #2" --body-plain="This is the body. 이건 바디다." --charset=UTF-8

When the email arrives in my gmail inbox, it contains the following UTF-8 encoded text in the body:

This is the body. 이건 바디다.


I originally heard about a similar Perl script called sendemail (not to be confused with the mail server sendmail), but it hasn't been updated since 2009 and currently doesn't work due to an SSL bug.

smtp-cli, on the other hand, is comparatively recent with version 3.6 updated July 11, 2013 and even has a package available through AUR. It works great!

2014년 7월 8일 화요일

MyRepos, aka 'mr' -- tool for managing multiple VCS repos

We have 10+ repos under ~/ and it is a real PITA to push all changes manually to our different remote repositories (some on Bitbucket, others on Github).

There is a great tool called MyRepos written by Debian Developer Joey Hess (well-known for debian-installer, debhelper, git-annex and lots of other popular packages) that allows you to manage multiple VCS repositories with a single command.

To setup mr to work with your repo, first you have to cd into the directory containing your repo and type:

mr register

so myrepos can make a note of the remote repo target.

Once that's done for all the repos you want to manage with mr, exit out of the directories under version control and run your VCS command prefaced by mr to update multiple repos at once. Here's an example of pushing local changes to 7 different remote git repositories:

[archjun@arch ~]$ mr push -u origin master
mr push: /home/archjun/Documents/MyProjects/KTokenizer
Counting objects: 3, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 475 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
   3986b86..541569e  master -> master
Branch master set up to track remote branch master from origin.

mr push: /home/archjun/Documents/WhatYoureReallyMeantToDo
Counting objects: 15, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (15/15), done.
Writing objects: 100% (15/15), 4.52 KiB | 0 bytes/s, done.
Total 15 (delta 6), reused 0 (delta 0)
   89e72c6..a55889f  master -> master
Branch master set up to track remote branch master from origin.

mr push: /home/archjun/Documents/cchallenges
Counting objects: 28, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (27/27), done.
Writing objects: 100% (28/28), 5.37 KiB | 0 bytes/s, done.
Total 28 (delta 10), reused 0 (delta 0)
   3158800..d810c82  master -> master
Branch master set up to track remote branch master from origin.

mr push: /home/archjun/Documents/coursera_IMT
Counting objects: 6, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 953 bytes | 0 bytes/s, done.
Total 6 (delta 4), reused 0 (delta 0)
   60e0628..987c4de  master -> master
Branch master set up to track remote branch master from origin.

mr push: /home/archjun/Documents/cs50x_2014
Counting objects: 44, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (42/42), done.
Writing objects: 100% (44/44), 4.75 KiB | 0 bytes/s, done.
Total 44 (delta 24), reused 0 (delta 0)
   03f6060..9cf44c8  master -> master
Branch master set up to track remote branch master from origin.

mr push: /home/archjun/Documents/idb_2014
Counting objects: 10, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (10/10), done.
Writing objects: 100% (10/10), 5.97 KiB | 0 bytes/s, done.
Total 10 (delta 5), reused 0 (delta 0)
   212016d..bf033fc  master -> master
Branch master set up to track remote branch master from origin.

mr push: /home/archjun/Documents/jun-project-euler
Username for 'https://github.com': gojun077
Password for 'https://gojun077@github.com': 
Branch master set up to track remote branch master from origin.
Everything up-to-date

mr push: finished (7 ok)

A real time-saver!

2014년 7월 1일 화요일

How to track Brainworkshop and Mnemosyne repetitions in Beeminder (using Dropbox and IFTTT)

I'm a huge fan of Beeminder as it has helped me quantify progress on personal goals. Before Beeminder, I just wrote down my goals on a To-Do list and then felt bad when I failed to achieve them.

With Beeminder, however, any quantifiable goal can be automatically tracked, and if I go off-track, my credit card is automatically charged a penalty that increases exponentially with every failure.

Although some users are conscientious enough to manually enter data points for their Beeminder goals, I am just too lazy to enter data every day. Also, I have found that manual data entry poses a greater temptation to falsify data points (especially if you have lots of money on the line). I find StickK along with a human referee much better for goals that cannot be automatically tracked (i.e. x pushups* per day, etc. -- for this type of goal, I take a video selfie and share it on social media with my referee).

*Postscript 2015.12.04
It actually is possible to track pushups through the Beeminder Android app using a method described on the Beeminder blog (Nov. 21) and in a post by the Beeminder CEO. Details:

By swiping at the data entry box, you can use different methods to submit data to Beeminder...
Tally entry
Swipe the data entry area right (see the narrow yellow rectangles?) and then tap the counter. Now you can increment the count by touching anywhere on the screen. I use this to do pushups by putting my phone on the floor and touching my nose to it on each pushup. Use the back button to submit the data.

Brainworkshop is a working memory trainer using Dual N-Back. Mnemosyne is a Spaced Repetition Software (SRS) program similar to Anki but more streamlined. Both are Free and Open Source Software.

First make sure that both Brainworkshop and Mnemosyne are installed into folders within Dropbox.

In Archlinux, Brainworkshop installs to ~/.brainworkshop by default, but you can create a symlink from this directory pointing to the actual Dropbox installation location (I go into detail of how to do this in this post).

Now create an IFTTT rule that uses RSS as the trigger. Dropbox conveniently provides each user with an RSS events feed that updates whenever a file is created or modified. Tell IFTTT to search this feed for the keyword .brainworkshop.stats

When this keyword is detected, make IFTTT send an email to bot@beeminder.com with your Beeminder ID and your goal name in the subject line in the following format:

User123/Mygoal555

In the body of the email, you want to increment your graph by one point, so the body of the email should read:

^ 1 "comment blah blah blah"

Note that there is a space between the caret and the 1. (I failed to notice this, and ended up spamming the Beeminder bot with a bunch of invalid emails; oops!)

*Edit May 2015:

IFTTT will soon add an official channel for Beeminder so it is no longer necessary to send an email to the Beeminder mailbot as in the above example if you activate the Beeminder channel and set your custom IFTTT recipe to increment a Beeminder goal by a certain amount every time your trigger condition is satisfied.

The IFTTT rule for Mnemosyne is much the same, except the keyword you want to search for in the Dropbox RSS Events feed is default.db-journal

--------------------------------------------
Bug Alert: Sometimes this doesn't work as intended, because if multiple files are changed at the same time, the Dropbox events feed will record the event as:

You edited filename.foo and n more files

instead of indicating a change in a single file like default.db-journal or brainworkshop.stats which the IFTTT trigger is looking for in the RSS feed.
--------------------------------------------

Here are the links to my shared IFTTT rules for Brainworkshop and Mnemosyne, respectively:

https://ifttt.com/recipes/159615-dropbox-brainworkshop-beeminder

https://ifttt.com/recipes/159617-dropbox-mnemosyne-beeminder

Feel free to edit these rules and replace the Dropbox RSS feed with your own. As of July 1st, 2014 I have been diligently doing my Brainworkshop and Mnemosyne reviews almost every day for 6 months straight!


2014년 6월 19일 목요일

As of Jun 19, 2014 Project Euler site is down :(

When I visited Project Euler on Thursday, I got the following page:

Apparently the site operators suspect their database was hacked. I hope they come back soon!

Update 2014-06-22: The site is back up, but logins are disabled and Project Euler banners listing how many problems a user has solved are no longer provided.

2014년 6월 17일 화요일

Google Code Jam 2014: First-ever Korean finalist Kim Chan-min (김찬민)!!!

Google Code Jam 2014 Round 3 was conducted last weekend (Jun 14) and the top 25 scorers will be going to LA to duke it out at the world finals along with the GCJ 2013 champion from Belarus.

Looking at the statistics of past Code Jams (previous years can be selected at the bottom of the page), there have been many finalists from Russia, Eastern Europe, China, Japan, America, etc., but no one from Korea (North or South) has ever made it to the onsite Final Round.

GCJ 2014 is thus a milestone of sorts for Korea, as Kim Chan-min kcm1700 placed 18th in the world in Round 3. Like most of his fellow qualifiers to the final round, he used C++ in his solutions. One of the reasons for the popularity of C++ in algorithmic coding competitions is that ACM-ICPC, which is the collegiate training arena for competitive coders, only accepts solutions in one of three languages -- C, C++, or Java -- and the International Olympiad in Informatics for high schoolers only accepts solutions in C, C++ or Pascal. This trend is also evident among the top competitors at elite algorithmic programming challenge sites like CodeForces and TopCoder.

So just who is kcm1700? He is a student at Korea's most prestigious college, Seoul National University and is a CodeForces grandmaster. According to his Twitter blurb, he participated in IOI 2008 and made it to the ACM-ICPC World Finals in 2010.

This new generation of programmers has a chance to change Korea's software development culture, which lags 15~20 years behind that of Japan and America. I won't go into the problems of Korea's software industry in this post, but I believe that the culture must change before Korea has even a remote chance of becoming a good environment for hackers (some good articles in Korean about the myriad problems faced by local SE's: 1 2 3).

Kudos to kcm1700 for making it to the GCJ 2014 World Finals and Good Luck!

Postscript 2014.09.25
kcm1700 scored 7th in the finals. A very respectable showing!

2014년 6월 10일 화요일

Handy Imagemagick tools for cropping book images prior to OCR

It's annoying to edit out page headers and footers that have been automatically OCR'd. One way to avoid this problem is to manually specify the OCR area, but this can become quickly tedious if you have hundreds of pages to process.

Enter Imagemagick's convert command. Today I will talk about the -chop and -shave config flags for convert.

These commands are a lifesaver when we want to unneeded areas (i.e. footers/headers, page numbers, etc.) that appear in a constant location from multiple pages.

A useful resource for the various image-cropping options available in convert can be found here:

http://www.imagemagick.org/Usage/crop/

Consider the following scan from a book:


Every page contains a page number at the bottom and on every other page along the right margin the book's title written vertically. We want to crop all the book's pages s.t. the page number and vertical title won't appear in the final image -- this will make OCR go much faster as we won't have to manually select which area needs to be OCR'ed.
In this particular example, we can use

convert "스님의_주례사 - 0011.png" -gravity SouthEast -chop 250x200 SE_chopped250x200.png

Which will remove 250 pixels from the right (East) and 200 pixels from the bottom (South). In the case of the -crop config flag, the reference point (0, 0) for all pixel calculations is the top-leftmost corner of the canvas. By using the -gravity flag, however (quote from Imagemagick docs):

The direction you choose specifies where to position text or subimages. For example, a gravity of Center forces the text to be centered within the image. By default, the image gravity is NorthWest.

So -gravity SouthEast will make the reference point (0, 0) the bottom-rightmost corner of the canvas. Now the resulting chopped image looks like:



​As you can see in the above image, the extraneous text from the right and bottom margins has been cropped out!

In other books, however, the location of extraneous text might be different. Let's say you want to remove text from both the top and bottom (or left and right) of the following screencap:

Eliminating the viewing frame can be accomplished with the convert flag -shave, which shaves pixels from the edges of an image (top & bottom, left & right). The arguments of -shave are:

... -shave [numPixelsLeftRightEdges]x[numPixelsTopBottomEdges]

Note that the brackets above should not actually be typed out. So if you wanted to remove 100 pixels from the left and right edges, you would pass the following arguments to - shave:

... -shave 100x0

If you want to remove 100 pixels from the top and bottom edges:

... -shave 0x100

To remove 100 pixels from both the top & bottom as well as left & right edges:

... -shave 100x100

The single page with its top and bottom edges removed:

convert escape_from_evil_frame_ex.png -shave 0x50 shaved_0x50.png


Finally, one more example. Say we have the following screencap containing two facing  pages:


Let's use the -shave flag to remove extraneous areas from both the top and bottom, left and right to make the image more amenable to OCR.

convert two_page.png -shave 80x50 two_page_shaved_80x50.png

This command shaves 80 pixels from both the left and right as well as 50 pixels from the top and bottom leaving us with the following image:


To run any of the above commands on all the images in a directory, simply invoke convert with a wildcard. For example:

convert *.png -configFlag outputFilename.png

Imagemagick will automatically increment outputFilename: outputFilename0.png, outputFilename1.png...

2014년 6월 1일 일요일

Overuse of English terminology in English-to-Korean technical translations

Recently I've been doing a technical translation project in the civil engineering and light rail sector. For those of you who have experience doing technical translation you will know that the biggest challenge in such projects is dealing with specialized terminology. In the engineering field common terminology might include terms such as

torque, leverage, pivot...

while in light rail projects common terms include

guideway, M&SF (Maintenance and Storage Facility), PSD (Platform Screen Door)...

and so on.

One problem I have noticed is that field engineers oftentimes transliterate these English terms to Korean, i.e. torque would become 토크 (which sounds like toe-keu for those of you who can't read Hangul). If you are an engineer who is familiar with the English term torque, than maybe transliteration (phonetic decomposition into another language) isn't such a big deal. But the problem is that engineering spec sheets and design review documents contain hundreds of such terms and in many cases, translators simply transliterate these to Korean without trying to find an appropriate term in Sino-Korean. As the final end-users of technical specification sheets include non-engineering personnel, naive transliteration that fails to convey meaning creates the risk of misunderstanding.

Looking up torque in Google using the search term: "define torque" returns:

1. MECHANICS - a twisting force that tends to cause rotation.

The Sino-Korean for this term is 회전력(回轉力) which literally means rotation(回轉) power(力).

In fact, most science and engineering terminology in Asia has Traditional Chinese character (aka hanja in Korean, kanji in Japanese) equivalents. The advantage of Chinese ideographs is that the actual principle explaining a word is contained in its symbolic representation. Even if you have no idea what torque means, if you look at its ideograph 回轉力 you can at least know that it has something to do with rotation. The ideograph is certainly more informative than the transliteration toe-keu (토그).

In Korea, there has been a systematic move away from the use of Chinese ideographs in favor of Hangul (native Korean script) over the past century and over the past 50 years or so, English transliteration into Hangul has become rampant.

But first, a historical detour into the background of Sino-Korean's decline in the Korean language. In 1894~1895, Korea's teetering Chosun dynasty passed the Second Gabo Reform (갑오개혁 甲午改革) which eliminated the civil service exam system (과거제도 科擧制度 ) based on the Chinese Classics while mandating the use of Hangul in lieu of Chinese characters in official documents. Fast forward 15 years to 1910, when Japan annexed Korea -- according to Japanese colonial gov't statistics (link in Korean), more than 90% of the Korean population could read neither Chinese nor Korean characters. (Note: the statistics from the colonial Japanese gov't should be taken with a grain of salt, as it was in their interest to discredit the natives as much as possible to justify their annexation) After Japan's defeat in WWII and Syngman Rhee's rise to power in South Korea, the illiteracy rate stood at 78% (link in Korean). Let me note that high illiteracy rates were common even in Western pre-industrial societies.

With the rise of nationalism in Korea's North and South, there was a movement to use pure Hangul free of Chinese characters. One problem with this plan, however, is that 70% of Korean originates from Chinese characters. The DPRK's solution was to remove many Sino-Korean words from the popular lexicon and replace them with pure Korean, a task at which they've been quite successful (one drawback, however, is that after 60 years of national division, people from the North and South sometimes have trouble communicating). South Korea, in contrast, pushed ahead with the Hangul-ization of school curricula, government publications, newspapers, etc. without removing Sino-Korean vocabulary from wide use.

This is problematic for one big reason -- different Chinese characters have lots of homophones in Korean. For example, the Chinese characters 鎭痛 (zhèn tòng, pain relief) and 陣痛 (zhèntòng, labor pains) have slightly different pronunciations in Chinese:



If you listen carefully to the Chinese pronunciation, you will notice that 鎭 has more stress on the first syllable than does 陣. Also the waveform display in the Soundcloud link above shows that the two are distinct. In Korean, however, the pronunciation for both 鎭痛 and 陣痛 is 진통 (jin-tong). Because these two different words are homophones, the only way you can know they mean different things is by using Chinese characters to disambiguate or by guessing from the context. But guessing from context is not foolproof. If you are a doctor in a busy hospital and you hear 'jin-tong' (진통), you might be confused as to whether it's referring to labor pains or the need for painkiller.

There are tens of thousands of such homophones in Korean and the meaning often cannot be gleaned from context alone. In conservative circles, there is much lament (link in Korean) about Korean language becoming a kind of Ebonics in which contemporary meaning and usage have been divorced from their linguistic roots.

I am no stickler for Korean tradition, but these critics have a point. Chinese ideographs can coexist with Hangul script for disambiguation purposes and provide more meaning than simple transliteration alone. Because of the growing role of English loan-words in Korean, reading comprehension is taking a hit. If Koreans were fluent English-speakers, then using such loan-words wouldn't matter, but I've noticed that my fellow translators often have no understanding of the technical material they're translating from English. This is exacerbated by the fact that most interpreters/translators in Korea are women, the vast majority of whom major in the humanities.

I believe this problem could be ameliorated somewhat if English-to-Korean technical translators used more Sino-Korean terminology, which includes semantics in their symbolic representation. Then translators could at least get a glimpse at the principles behind the terms they're translating.

Postscript
Some English engineering terms, their naive transliterations into Korean, and Sino-Korean equivalents:

English Transliterated Sino-Korean
torque 토크 회전력(回轉力), 선전력(旋轉力)
actuator 엑추에이터 작동기(作動機)
leverage 레버리지 공간력(槓杆力)
buffer 버퍼 완충장치(緩衝裝置)
detailing 디테일 세부장식(細部裝飾)