Aiaioo Labs

This is an introduction to Vaklipi, a system that lets you write programs and give instructions to a computer using English, Hindi, Kannada, Tamil, French, Chinese, etc.

The Java command System.out.println(3×4+2) returns the answer 14. The same thing in Vaklipi would be What is 3 into 4 plus 2?

In German, it would be Was ist 3 mal 4 addiert zu 2?

1. Use double quotes to quote something, not single quotes.
2. Write numbers as roman numerals not as words.
3. It doesn’t yet get coreferences (words like ‘it’ and ‘them’). Say ‘and increment x’, not ‘and increment it’.

• Japanese & Chinese: Only pinyin & romaji are supported. Write と as ‘tho’.
• Information: There is an infinite loop arrester present, with a limit of 1000 loops.
• Limitations: The demo only supports 2 datatypes – numbers and strings. There are more on the way.

If you’re a beginner to programming, start at one of the following:

1. Introduction: Learn about Vaklipi
2. Output: Learn to write out messages in Vaklipi

Join this community mailing list for announcements:

The zeroth thing you need is the Pocketsphinx binaries.

Just download the win32 binaries from the Sphinx website (download pocketsphinx, sphinxbase, sphinxtrain and cmuclmtk from the Sphinx website).

The first thing you need to do is build a language model or a grammar.

The grammar can be something simple in a format called JSGF, and this is the easier way to get a speech recognizer up and running. Alternatively, you can use a language model. The language model can be built using the instructions on the Sphinx site. You can create it starting from a file with sentences like this:

<s> I WANT A NEXTCUBE ZERO FOUR ZERO </s>
<s> I WANT THE NEXTCUBE ZERO FOUR ZERO </s>
<s> I NEED A NEXTCUBE ZERO FOUR ZERO </s>
<s> I NEED THE NEXTCUBE ZERO FOUR ZERO </s>
<s> I AM LOOKING FOR A NEXTCUBE ZERO FOUR ZERO </s>
<s> I AM LOOKING FOR THE NEXTCUBE ZERO FOUR ZERO </s>
<s> I AM SEEKING A NEXTCUBE ZERO FOUR ZERO </s>
<s> I AM SEEKING THE NEXTCUBE ZERO FOUR ZERO </s>

A sample JSGF file would be (modified from the sample on the Sphinx website) … note that I’ve made all the words capitals because the CMU phonetic dictionary has all the words listed in caps (make sure that any language model is all caps as well, except for the sentence boundaries):

#JSGF V1.0;
/**
 * JSGF Grammar for Hello World example
 */
grammar hello;
public <greet> = (GOOD MORNING | HELLO | HI) ( PAUL | RITA | WILL );

The second thing you need is an Acoustic Model

An acoustic model maps sound features from the speech recognizer to phonemes.
Voxforge provides a free acoustic model for Pocketsphinx that you can use.

The third thing you need is a phonetic dictionary

The phonetic dictionary maps the recognized phonemes to actual words in your language. For English, there is a phonetic dictionary available from CMU

You will just need to download one file: cmudict.0.7a_SPHINX_40

Now, you have all the components you need!

Running Pocketsphinx

With JSGF:

$ pocketsphinx-0.7-win32/pocketsphinx_continuous.exe \
-hmm voxforge-en-r0_1_3/model_parameters/voxforge_en_sphinx.cd_cont_3000 \
-jsgf greet.jsgf \
-dict cmudict.0.7a_SPHINX_40

With a language model:

$ pocketsphinx-0.7-win32/pocketsphinx_continuous.exe \
-hmm voxforge-en-r0_1_3/model_parameters/voxforge_en_sphinx.cd_cont_3000 \ -lm cmuclmtk-0.7-win32/output.lm.DMP \
-dict cmudict.0.7a_SPHINX_40l

Any additional phonetic entries in the phonetic dictionary can be created using the CMU dictionary phoneme set

Education

1. Videos on speech recognition
2. Lectures on speech recognition
3. Voxforge has an article on what an acoustic model is

Imagine a day when spoken language sentences can be recognized perfectly by a machine.

It suddenly becomes much easier and more natural to issue commands and pose queries in a natural language, rather than, say in XML. Imagine speaking a C program to someone, syntax and all. That would be really weird!

We recently evaluated spoken language programming by combining a speech recognizer with Vaklipi, our fifth generation programming language.

The results were frankly disappointing.

Here’s a sample conversation between me and my computer:

Cohan : b is equal to two.
Sphinx hears: b is e equal to two.
Vaklipi [error]: I can’t understand …
Cohan : b equals two.
Sphinx: b equals two.
Cohan : What is b.
Sphinx: What is 8.
Vaklipi : 8.0
Cohan : What is b.
Sphinx: What is b.
Vaklipi: 2.0

I think I’ll wait a few more years ;)

Thanks to everyone who filled out the long and tedious questionnaire to help us evaluate the algorithms used in Vaklipi for a research paper.

The paper we submitted to IJCNLP in 2011 titled Natural Language Programming Using Class Sequential Rules was accepted to the conference, and so we traveled to Chiang Mai in Thailand in November 2011 to make an oral presentation of the paper to our peers.

Here is a demonstration of Vaklipi as a programming tool.

It contains ‘Fibonacci Number Generator’ programs in Hindi, Kannada and Tamil.

There are millions of students who study mainly in these languages, and they might some day be able to learn programming using systems such as VakLipi.

Here’s a demo of Vaklipi as a tool for translating math problems into other languages

Thanks to all the people filled out the surveys we sent out and helped evaluate Vaklipi. We also thank the people who helped us port the system to other languages like: Kannada (Ms. K. G. Padma Lekha, Mr. K. G. Srikanta Dani, Dr. K. R. Ganesha, and Mr. Rupesh Kumar G.), Tamil (Mrs. Linda Christy and Dr. S. Carlos), Hindi (Mr. Kartik Asooja of Aiaioo Labs and Mr. Chandra Bhan Asooja), French (M. Sammy Ben Rabah, M. Yann Jouanique et Mme. Fanny Jouanique), German (Hartmut Wege, Judith Klein und Deepica Rao), Chinese (under development – Dright Ho), Japanese (Jojo Baby), Polish (under development – Joanna Lupinska – Asia), Telugu (under development – Mrs. Meenakshi Jami).