A Glimpse into Foreign-Language News

Although the language of global communication is (still) English, there's lots of interesting content in other languages too. What's a curious English speaker to do? For example: A Bulgarian news site may only publish in Bulgarian because the target audience is in Bulgaria. Or: A website with international readers, like Al Jazeera, publishes in both Arabic, the region's native language, and English. But the English articles may not be exact translations, and native-language articles aimed at a domestic audience may not be translated. Wouldn't it be interesting to get at least a glimpse into what the domestic audience is reading?

One option for reading articles from foreign-language websites already exists: Watching America. It publishes articles about the US, found on non-US news sites, and translated to English by volunteers.

I had something else in mind: An overview of any articles on foreign-language sites, whether about the US or not, thus including domestic news as well. My demo, with the (admittedly awkward-to-type) name of What's New(s)?, is my first attempt. It retrieves the titles and summaries of news articles from selected foreign-language news sites, and translates the content to English using AI. The summaries are updated several times a week.

How It Works

The website has two functions: Behind the scenes, it fetches, translates and saves article summaries periodically. Independently, when someone visits the site, the most recent article titles and summaries are displayed on the What's New(s)? home page.

Fetch, Translate and Save the Content

The first function consists of three steps:

1. Fetch the foreign language content. Many sites offer an RSS feed, a file that contains summaries and URLs for recently-published content. Another way to fetch articles is using a web scraper.

2. Translate the title and summary to English. This is done via the DeepL API to translate texts instantaneously.

3. Store the results in the What's New(s)? database. We don't want to fetch and translate the articles every time someone visits the site, so we save the original and translated content in a database.

Display the Articles

Once the original and translated content has been stored in the database, surfing over to the What's New(s)? home page will display the article titles and summaries.

English translations are displayed by default. To see the original foreign-language texts, select the original language from the pull-down menu for each site. Click on the go button at the bottom of each summary to take you to the original foreign-language article.

(Click on a thumbnail to enlarge.)

The Demo

Here's the demo, with a few hand-picked foreign news sites: whatsnews.jbrcodes.com. (Full disclosure: non-Latin alphabets fascinate me.)

Notes

• RSS Feed Versus Web Scraping: If a foreign-language website offers an RSS feed, that is the easier solution; the feed provides all the information we need: title, summary, publication date, and URL of the article. The disadvantage of RSS feeds is that the order of the articles is typically reverse-chronological, whereby the first article is the one most recently published, not necessarily the most newsworthy. In contrast, a web scraper allows us to choose the headlines from the home page. The disadvantage is that it requires analyzing the foreign-language website's home page, and writing a small program to periodically extract the desired information.

• AI-based Language Translation: Computer translation works great in some situations, but it isn't perfect, and it possibly never will be. For the purposes of this demo, AI translation works fine. But human translation is probably still the better option for texts requiring more nuance.

Technologies Used

As with all my demos, What's New(s)? is built on the proverbial shoulders of existing powerful tools:

DeepL

An API to translate texts using AI.

feedparser

A Python module/extension to download and analyze RSS files.

Flask

A Python-based web server framework.

Peewee

A database ORM used to manage the original and translated texts in a relational database.

Playwright

A tool used here for web scraping, although it is possibly better known for website testing.

SQLite

A smaller and simpler database alternative to networked relational databases (like MS-SQL, MySQL, PostgreSQL) that is perfectly adequate for many low- to medium-traffic websites.