Category: python

Swahili text normalizer

One of the challenges with working with text corpus, that will be used for voice datasets, is that you will encounter non-standard text like percentages, numbers, time and date. These can be a challenge when one has to read them. For example 23 could be read as twenty three or two three. To fix such an issue you do what is called text normalization during the cleaning and validation stage for your text corpus. This is helpful especially when your text corpus will be used for speech use cases.

Text normalization in speech applications converts written, non-standard text (like numbers, symbols, and abbreviations) into spoken-word forms for text-to-speech (TTS), or structures raw audio transcripts into clean data for automatic speech recognition (ASR)

During my fellowship with Mozilla Foundation working on the common voice text corpus for Kiswahili, this is something I encountered during the sentence validation stage. On the common voice platform, one way one can contribute to building the Swahili voice dataset is to read sentences in the Swahili language. The sentences went through a validation stage before they were ready for reading. You can see that encountering something like 3.5% might slow the reader (silence in the voice clip making it longer) or present a challenge of how to read the percentage. For English you can use already existing natural language processing tools to do text normalization.

I spent some time working on a small library for the Swahili language. Its a small collection of utilities that convert:

Numbers to words (0 .. < 1e9)
Years, dates, times to words
Decimals, percentages to words

You can find the source code here https://github.com/bmwasaru/kiswahili-text-normalizer and the readme file has instructions on how to install and use it. Here is an example usage:


>>> import kiswahili_text_normalizer as ktn
>>> text = "Tutakutane 20/11/2025 14:30, malipo ni 12.5%."
>>> print(ktn.normalize_text(text, profile="asr"))
tutakutane tarehe ishirini mwezi wa kumi na moja mwaka elfu mbili ishirini na tano 
saa nane na nusu malipo ni asilimia kumi na mbili nukta tano

As you can see it converts

20/11/2025 to tarehe ishirini mwezi wa kumi na moja mwaka elfu mbili ishirini na tano
14:30 to saa nane na nusu
12.5% asilimia kumi na mbili nukta tano

You will note it adds keywords like tarehe to denote date, saa to denote time, nukta to denote decimal point and asilimia to denote percentage.

I have used this on text that I had scrapped from the Voice of America Swahili news website before it shutdown. The code for that can be found here https://github.com/bmwasaru/voa/blob/main/scripts/normalize_text.py. This was a perfect use case because this was a large text corpus and the normalizer was able to quickly find and convert. Side note, if you are looking for Swahili language text you can find the scraped text in cvs files here https://github.com/bmwasaru/voa/tree/main/sentences.

I hope one finds this useful and please share some feedback 🙂

python TILs

secrets.randbelow(x)

TIL (Today I learned) about the python programming language module secrets that can be used to generate random numbers. You can generate random integer below a certain upper bound, say we can generate a number below 200 as follows:

			
import secrets
print(secrets.randbelow(200))

So how did I come across this? I was looking at how the Hushline (https://hushline.app), a whistleblower platform that provides secure and anonymous tip lines, was handling authentication. They use the secrets module to generate two random numbers for the CAPTCHA math question to verify if a human and not a bot is trying to authenticate. You can look at the code here https://github.com/scidsg/hushline/blob/main/hushline/routes/auth.py#L62. They use the secrets library to generate two random numbers below 10 then add 1 to the numbers that will be displayed to the human to do the math.

python

nltk.download

Was working on some sentences and needed to download the ‘punkt’ nltk (natural language toolkit) data file but encountered the following issue:

[nltk_data] Error loading punkt: <urlopen error [SSL:
[nltk_data]     CERTIFICATE_VERIFY_FAILED] certificate verify failed:
[nltk_data]     unable to get local issuer certificate (_ssl.c:1000)>

This seemed to be a ssl issue. So I run this command:

sh "/Applications/Python 3.12/Install Certificates.command"

Which installed certifi

-- pip install --upgrade certifi
Collecting certifi
  Using cached certifi-2024.7.4-py3-none-any.whl.metadata (2.2 kB)
Using cached certifi-2024.7.4-py3-none-any.whl (162 kB)
Installing collected packages: certifi
Successfully installed certifi-2024.7.4

Then attempted download of the nltk data files via the terminal:

python3 -c ""import nltk;nltk.download('punkt')"

heroku python

Heroku Database Credentials

Post author By bmwasaru
Post date December 10, 2020
No Comments on Heroku Database Credentials

When deploying to Heroku and use a database it creates the DATABASE_URL environment variable for you that you can connect your application with.

With python you can access this using the os module as follow:

os.environ.get('DATABASE_URL')

Though you might need to access the different details like; database name, user and password. Here is how I get to access this details, in this example I am using ClearDB, MySQL host provider.

import os


if os.environ.get('CLEARDB_DATABASE_URL'):
    db_url = os.environ['CLEARDB_DATABASE_URL']
    mysql_database = db_url.split('/')[3]
    mysql_user = db_url.split('//')[1].split(':')[0]
    mysql_password = db_url.split('@')[0].split(':')[2]
    mysql_host = db_url.split('@')[1].split('/')[0]
else:
    mysql_database = ""
    mysql_user = ""
    mysql_password = ""
    mysql_host = "localhost"

Follow me on twitter here https://twitter.com/bmwasaru