Local language is critical for e-commerce companies to expand in smaller towns and rural areas. In August 2020, Flipkart forged a partnership with IIT-Patna to automate the process of translating and transliterating over 5.4 million words from English to Indian languages. That joint work has helped Flipkart to today offer their app interface in Hindi, Kannada, Tamil, Telugu, Malayalam, Marathi, Bengali, Gujarati, Odia, Assamese, and Punjabi.
“In Q4 of 2020, Flipkart’s vernacular user share was 12%. Just one year after that, it had increased to 18%,” says Mayur Datar, Flipkart’s chief data scientist.
The content on Flipkart is largely generated by sellers and buyers. The queries and product reviews are riddled with grammatical and spelling errors, as well as a lot of noise and emotion. “Another major issue is that they appear to be in English, but more than half of the words are in other languages. A query could be, ‘Barah saal ki ladki ke shoes.’ ‘Awesome’ is often ‘osm’. For such unstructured sentences, generic machine translation services like Google and Microsoft are largely ineffective. This is where our collaboration with IIT-Patna has been extremely beneficial. They provided us with out-of-the-box prototypes for translating complex text,” says Datar.
Asif Ekbal, associate professor in the department of computer science and engineering at IIT-Patna, and who heads the research team, says his institution is the leader in text, video and speech analysis. “Flipkart provided us with real-time data to work with, and we also got to explore grey areas like code-mixing (mixing of two or more languages in speech),” he says.
Unlike in Europe, where translations have been going on for years even before the advent of deep learning, there is a huge scarcity of parallel corpora (collection of original texts in a language and their translation into a set of languages) in India, particularly in the product review domain.
Flipkart began by translating the vast amount of data available in English – search queries, user reviews, product descriptions – into Hindi, using available technology such as deep learning, to create parallel texts. The IIT-Patna team devised filtering techniques to segregate this rudimentary translation into three categories – good, could be better, and bad – and bias the synthetic data to be more colloquial and capture sentiments. Humans were used only to improve poor language pairs. “We didn’t have to manually translate millions of sentences, but only a few thousands,” Datar explains.
IIT-Patna researchers used data augmentation techniques to gradually increase the size of the synthetic data. They used a cross-lingual embedding technique to create a neural machine translation system that could translate code-mixed sentences. In cases where a word in the source language had multiple meanings in the target language, the team devised a constant decoding method to train the ML system to place the right words in the right context.
Based on the success of the Hindi app, the collaboration expanded the prototypes to other Indo-Aryan languages, followed by Dravidian languages. And the project is ongoing. (Source: Times of India)
