Welcome

This website is a multi-language CMS / Blog I build with Python, the framework Flask and SQLAlchemy. It is deployed on my ISPConfig3 (Nginx + Mariadb) server using Docker which is also used for development. Read more about this website.

Coronavirus

I investigated the deaths in The Netherlands in 2020 and looking at the data, I can only wonder why we have never had, and still do not have, a serious discussion about reopening the economy for people under 65 or, for example, people under 50. Read the findings and conclusions.

Recent

Prevent IP address spoofing using Reverse path filtering

15 June 2023 0

Peter

This post is about system administration and has nothing to do with Python. Then why post this here? Because I believe there many like me who run one or more web servers and sometime run into these problems. In the previous post I wrote that my ISPConfig Debian server was subject to port scanning, etc. and that it appeared that 95% of all requests came from China unless ... these IP addresses were ...

Collect and block IP addresses with ipset and Python

21 May 2023 0

Peter

If you have a server connected to the Internet, you've probably seen this in your log files: lots of illegal external requests trying to access your services. I maintain a Debian server, and use Fail2Ban for intrusion prevention. Standard practice, install, configure and forget. Since the server was getting pulled down at certain times, I decided to take a closer look. I'm mostly into programming ...

How to cancel tasks with Python Asynchronous IO (AsyncIO)

2 May 2023 0

Peter

For a project I was using AIOHTTP to check the responses of many remote websites, URLs. The URLs were coming from a list. This list can contain duplicates. Everything fine until I noticed that some responses also had status code: HTTP-429 'Too Many Requests'. Whatever the reason, overload, security, we want to behave friendly and do not want to call identical URLs again, at least for a minimum time. ...

Run a Docker command inside a Docker Cron container

18 April 2023 0

Peter

When using Docker, your application typically consists of several Docker containers. Often, you want to run scripts inside these containers at certain moments, for example, every five minutes, once an hour, once a day. This is where the job scheduler Cron comes in, and there are several options on how to do this. In this post I create a separate Cron container, and use the Docker Exec command to execute ...

Creating a Captcha with Flask, WTForms, SQLAlchemy, SQLite

10 April 2023

Peter

In the past I wrote some code for a (text-only) Captcha for a Flask website. This is an update. Here I use a Pypi package to generate the image. Besides that I also added a Captcha image refresh button.You can try the code below, it is a (newsletter) subscription form. Although this is a Captcha solution for a Flask website, it can be converted into a Captcha server. Note that I use SQLite here for ...

Multiprocessing, file locking, SQLite and testing

30 March 2023 0

Peter

I was working on a project with SQLAlchemy and PostgreSQL. For a few tables, I wanted to limit the number of rows per user, and did this by adding a PostgreSQL check function and trigger. Manual testing every thing appeared to be working fine but what if a user would start multiple processes and add rows at exactly the same time? I added the 'pg_advisory_xact_lock' but will this really work? Did I ...

Sending messages to Slack using chat_postMessage

23 March 2023 0

Peter

For a project I was already sending messages by email, but now I also wanted to send messages to Slack. Of course, we are using the Python Slack SDK. The documentation can be found on the page: Python Slack SDK - Web Client. In this post I create a simple SlackAPI class with its own SlackError exception class. Create and configure a new Slack App We will be sending our messages to a Slack Channel. ...

PostgreSQL backup with Docker SDK for Python

9 March 2023 0

Peter

This is a short post about backup of a Dockerized PostgreSQL database. To access the database, we typically run a Bash script on the host, with commands like: docker exec -t <container> bash -c '<command>' In this post we replace our Bash script with a Python script. Why? Because we know Python and programming in Bash can be time-consuming. While we can use 'subprocess' to run all system ...

Prevent sending duplicate messages to a remote system

21 February 2023 0

Peter

Many times applications must send messages to a remote system. In a perfect world we only have to deal with the Happy Flow: There are no bad things happening like exceptions, errors. Unfortunately, the world is not perfect. Besides programming errors, connections can fail, database systems can fail, remote systems can fail. A simple question - can you write some code to send messages to a remote system ...

Politician Translator with Spacy and Negate

11 January 2023 0

Peter

This is a short post. All the time we hear these politicians talking but most of the time they mean the opposite. For example, if a politician says that he will lower the taxes, the taxes will go up. If a politician says he did not a relationship with that woman, then ... Etc. So I thought, why not make a Politician Translator in Python? In this post, I start with the results. The code is at the end. The ...

From monolithic code to services with RabbitMQ and Pika

8 January 2023 0

Peter

This post is about using RabbitMQ in your Python application. If you are using Rabbit already, you will probably find nothing useful in this post. Why a post about RabbitMQ? Because I have an application using it.It is running for a year, and I thought to share my experiences. In this post, we transform a monolithic application into services decoupled by queues. Also, I will talk about services instead ...

Flask application showing stdout and stderr of a background job

19 December 2022 0

Peter

In a Flask project, I needed to run a background job, more specifically a command running in a (Linux) terminal, and show its output, stdout and sterr, in real time in a browser window. You can find some solutions on the internet and this is just another one. I am also using some code I found on the web, see links below. This solution is using: multiprocessing, to start a new process from our Flask ...

Aggregate and tail Docker container logs using Docker SDK for Python

11 December 2022 0

Peter

The problem: You finally have a Docker application consisting of many (micro) services (containers) and want to monitor all these containers for errors. The services are mostly Python scripts that use the standard Python logging module and print messages to stdout (and stderr): ... DEBUG ... ... INFO ... ... ERROR ... Using the default Docker json-file logging driver, these messages end up in ...

Python application logging with Docker

5 December 2022 0

Peter

When you develop a software application, probably the first thing you will set up is logging. First only to the console, but soon you will add log files. Some years ago I started using Docker for development and production. I did not make much changes when moving to Docker. The application log files are still in a log directory on a Docker Volume. This means that the log files are part of the application ...

LogLineFollower: Follow lines of a growing log file

9 September 2022 0

Peter

I was looking for a way to process lines of a log file while it was growing. I found some snippets and packages on the internet but not exactly what I wanted. That is why I decided to write my own. After having coded a first version, I searched again and found several more packages. But, looking at the description, the code and issues, I decided to stick with my own code. It cannot be that hard to ...

Connect to a service on a Docker host from a Docker container

11 August 2022 2

Peter

If you have some Docker problem and search the Internet, you will almost certainly stumble upon the question: How can I connect to localhost? What people mean is: How can I connect to a service on the Docker host from a Docker container. When I started using Docker I was also struggling with this. I am using Linux, Ubuntu, and when finally host.docker.internal was available for Linux many people thought ...

AIOHTTP: Detecting DNS timeout with custom nameservers

27 July 2022 0

Peter

When using AIOHTTP to fetch data from a web page on the internet you are probably using a timeout to limit the maximum waiting time. If you are using a domain name then the IP Address must be resolved. Without using a separate resolver you are dependent on the underlying operating system. Any errors propagate to your application. I did not want this dependency and specify the nameservers myself, ...

Flask Message Flashing: Replace Bootstrap Alerts by Toasts

25 July 2022 0

Peter

When you have a Flask application with Bootstrap, you are probably using Bootstrap Alerts to show flashed messages. I use them, and they work, but I am not really happy. By default, they do not look nice and in most cases they take a lot of space on the screen. And do you really want notifications like 'you are logged in' to be a Bootstrap Alert that must closed by the user? By adding a timeout we ...

SQLAlchemy: Using Cascade Deletes to delete related objects

16 July 2022 3

Peter

Although the documentation of SQLAlchemy ORM is very good it sometimes is confusing and lacks clear examples, but maybe this is just me. About Cascade Deletes, I find the following text from the SQLAlchemy documentation mind-blowing: The default value of the relationship 'cascade'-option is 'save-update, merge'. The typical alternative setting for this parameter is either all or more commonly 'all, ...

SQLAlchemy PostgreSQL: Add a second BigInteger Primary Key

28 May 2022 0

Peter

Suppose we use UUID Primary Keys in our project. This works fine, but there are also a few tables with millions of records. We know that SELECT and INSERT operations with UUID-based Primary Key tables can be much slower compared to Integer based Primary Key tables. What we want is for these huge tables to have only BigInteger Primary Keys and link them to other UUID Primary Key based tables, ...

Welcome

Coronavirus

Recent

Most viewed

Tags

Prevent IP address spoofing using Reverse path filtering

Collect and block IP addresses with ipset and Python

How to cancel tasks with Python Asynchronous IO (AsyncIO)

Run a Docker command inside a Docker Cron container

Creating a Captcha with Flask, WTForms, SQLAlchemy, SQLite

Multiprocessing, file locking, SQLite and testing

Sending messages to Slack using chat_postMessage

PostgreSQL backup with Docker SDK for Python

Prevent sending duplicate messages to a remote system

Politician Translator with Spacy and Negate

From monolithic code to services with RabbitMQ and Pika

Flask application showing stdout and stderr of a background job

Aggregate and tail Docker container logs using Docker SDK for Python

Python application logging with Docker

LogLineFollower: Follow lines of a growing log file

Connect to a service on a Docker host from a Docker container

AIOHTTP: Detecting DNS timeout with custom nameservers

Flask Message Flashing: Replace Bootstrap Alerts by Toasts

SQLAlchemy: Using Cascade Deletes to delete related objects

SQLAlchemy PostgreSQL: Add a second BigInteger Primary Key