angle-uparrow-clockwisearrow-counterclockwisearrow-down-uparrow-leftatcalendarcard-listchatcheckenvelopefolderhouseinfo-circlepencilpeoplepersonperson-fillperson-plusphoneplusquestion-circlesearchtagtrashx

Welcome

This website is a multi-language CMS / Blog I build with Python, the framework Flask and SQLAlchemy. It is deployed on my ISPConfig3 (Nginx + Mariadb) server using Docker which is also used for development. Read more about this website.

Coronavirus

I investigated the deaths in The Netherlands in 2020 and looking at the data, I can only wonder why we have never had, and still do not have, a serious discussion about reopening the economy for people under 65 or, for example, people under 50. Read the findings and conclusions.

Recent

Most viewed

Tags

AiohttpAJAXAlembicAlpineAPIApplication settingsAsyncBabelBeautifulSoupBootstrapCachingCaptchaCeleryColorsCompileCryptographyCSRF protectionCythonDecoratorDeep LearningDeepLDispatcherMiddleWareDistributionDNSDockerDocker-composeEmailExceptionsFastAPIFlaskGunicornIconsIMAPInternetISPConfigJavascriptJinja2Log fileLoggingMachine LearningMariaDBMessage brokerMultilanguageMultiprocessingPentestingPikaPostfixProfilingPyInstallerpyOpenSSLPytestRabbitMQRedisSecurity testingSQLAlchemyTestingThreadsTimezoneToastsUbuntu TouchUUIDWerkzeugWheelsWTForms

PostgreSQL backup with Docker SDK for Python

9 March 2023 0

Peter

post main image This is a short post about backup of a Dockerized PostgreSQL database. To access the database, we typically run a Bash script on the host, with commands like: docker exec -t <container> bash -c '<command>' In this post we replace our Bash script with a Python script. Why? Because we know Python and programming in Bash can be time-consuming. While we can use 'subprocess' to run all system ...

Prevent sending duplicate messages to a remote system

21 February 2023 0

Peter

post main image Many times applications must send messages to a remote system. In a perfect world we only have to deal with the Happy Flow: There are no bad things happening like exceptions, errors. Unfortunately, the world is not perfect. Besides programming errors, connections can fail, database systems can fail, remote systems can fail. A simple question - can you write some code to send messages to a remote system ...

Politician Translator with Spacy and Negate

11 January 2023 0

Peter

post main image This is a short post. All the time we hear these politicians talking but most of the time they mean the opposite. For example, if a politician says that he will lower the taxes, the taxes will go up. If a politician says he did not a relationship with that woman, then ... Etc. So I thought, why not make a Politician Translator in Python? In this post, I start with the results. The code is at the end. The ...

From monolithic code to services with RabbitMQ and Pika

8 January 2023 0

Peter

post main image This post is about using RabbitMQ in your Python application. If you are using Rabbit already, you will probably find nothing useful in this post. Why a post about RabbitMQ? Because I have an application using it.It is running for a year, and I thought to share my experiences. In this post, we transform a monolithic application into services decoupled by queues. Also, I will talk about services instead ...

Flask application showing stdout and stderr of a background job

19 December 2022 0

Peter

post main image In a Flask project, I needed to run a background job, more specifically a command running in a (Linux) terminal, and show its output, stdout and sterr, in real time in a browser window. You can find some solutions on the internet and this is just another one. I am also using some code I found on the web, see links below. This solution is using: multiprocessing, to start a new process from our Flask ...

Aggregate and tail Docker container logs using Docker SDK for Python

11 December 2022 0

Peter

post main image The problem: You finally have a Docker application consisting of many (micro) services (containers) and want to monitor all these containers for errors. The services are mostly Python scripts that use the standard Python logging module and print messages to stdout (and stderr): ... DEBUG ... ... INFO ... ... ERROR ... Using the default Docker json-file logging driver, these messages end up in ...

Python application logging with Docker

5 December 2022 0

Peter

post main image When you develop a software application, probably the first thing you will set up is logging. First only to the console, but soon you will add log files. Some years ago I started using Docker for development and production. I did not make much changes when moving to Docker. The application log files are still in a log directory on a Docker Volume. This means that the log files are part of the application ...

LogLineFollower: Follow lines of a growing log file

9 September 2022 0

Peter

post main image I was looking for a way to process lines of a log file while it was growing. I found some snippets and packages on the internet but not exactly what I wanted. That is why I decided to write my own. After having coded a first version, I searched again and found several more packages. But, looking at the description, the code and issues, I decided to stick with my own code. It cannot be that hard to ...

Connect to a service on a Docker host from a Docker container

11 August 2022 0

Peter

post main image If you have some Docker problem and search the Internet, you will almost certainly stumble upon the question: How can I connect to localhost? What people mean is: How can I connect to a service on the Docker host from a Docker container. When I started using Docker I was also struggling with this. I am using Linux, Ubuntu, and when finally host.docker.internal was available for Linux many people thought ...

AIOHTTP: Detecting DNS timeout with custom nameservers

27 July 2022 0

Peter

post main image When using AIOHTTP to fetch data from a web page on the internet you are probably using a timeout to limit the maximum waiting time. If you are using a domain name then the IP Address must be resolved. Without using a separate resolver you are dependent on the underlying operating system. Any errors propagate to your application. I did not want this dependency and specify the nameservers myself, ...

Flask Message Flashing: Replace Bootstrap Alerts by Toasts

25 July 2022 0

Peter

post main image When you have a Flask application with Bootstrap, you are probably using Bootstrap Alerts to show flashed messages. I use them, and they work, but I am not really happy. By default, they do not look nice and in most cases they take a lot of space on the screen. And do you really want notifications like 'you are logged in' to be a Bootstrap Alert that must closed by the user? By adding a timeout we ...

SQLAlchemy: Using Cascade Deletes to delete related objects

16 July 2022 1

Peter

post main image Although the documentation of SQLAlchemy ORM is very good it sometimes is confusing and lacks clear examples, but maybe this is just me. About Cascade Deletes, I find the following text from the SQLAlchemy documentation mind-blowing: The default value of the relationship 'cascade'-option is 'save-update, merge'. The typical alternative setting for this parameter is either all or more commonly 'all, ...

SQLAlchemy PostgreSQL: Add a second BigInteger Primary Key

28 May 2022 0

Peter

post main image Suppose we use UUID Primary Keys in our project.  This works fine, but there are also a few tables with millions of records. We know that SELECT and INSERT operations with UUID-based Primary Key tables can be much slower compared to Integer based Primary Key tables. What we want is for these huge tables to have only BigInteger Primary Keys and link them to other UUID Primary Key based tables, ...

Redirect on an exception in Flask using a decorator

7 May 2022 0

Peter

post main image In a Flask application, you typically implement global exception handlers. In many cases, this is sufficient. But what if you want more control? In one project, I was connecting to an API and I wanted a number of routes that used the API to redirect to a 'start' page in case of an API error, with an appropriate message of course. I implemented this using a 'redirect_decorator' exception handler that ...

SQLAlchemy Many-To-Many: Four ways to select data

12 April 2022 0

Peter

post main image This is a short post about Many-To-Many selection with SQLAlchemy. In the past I used the association (link) table in ORM queries, because I thought it must be fastest. This time I did a small test comparing various ways to select data. Spoiler: The association (link) table way is (of course) fastest. The model We have a Many-To-Many relationship between Orders and Products. # link table: order ...

Testing the RabbitMQ Pika publishing examples

25 March 2022 0

Peter

post main image A lot has been written about synchronous vs asynchronous publishing with RabbitMQ, see links below, I am not going to repeat this here. As this is my first time using RabbitMQ, I wanted to try both the synchronous and asynchronous publishing versions, using examples included with Pika, the RabbitMQ (AMQP 0-9-1) client library for Python. To my surprise, the Pika asynchronous example did not detect ...

An attempt to solve Tic-Tac-Toe using Keras and LSTM

2 March 2022 0

Peter

post main image After implementing my first Deep Learning LSTM model for a project I was thinking if Deep Learning could also solve a game. The first game that comes to mind is Tic-Tac-Toe. Then you search the internet and there appear to be many many people who had the same idea. Of course. Below I present my solution to solve Tic-Tac-Toe using Keras and LSTM (Long Short Term Memory). This is Deep Learning and not ...

LSTM multi-step hyperparameter optimization with Keras Tuner

13 February 2022 0

Peter

post main image A previous post was about Hyperparameter optimization with Talos. I could not get this to work with my LSTM model for univariate multi-step time series forecasting, because of the 3D input, so I switched to Keras Tuner. In this post I try to predict the next period of a sine wave using the Hyperband tuning algorithm. To reduce tuner time, I reduced the number of hyperparameters that can be optimized ...

Finding the closest matching sentence from a list of sentences

6 February 2022 0

Peter

post main image For a project I was looking for a way to match an incoming sentence with a list of fixed sentences. This is a complex subject but when searching the internet I bumped on the amazing project Sentence-Transformers. I modified one of the examples on this site to use a text file with sentences and typed some input sentences as a demo. Then I downloaded a text file from the internet 'Choosing a Cat, by ...

Automating Keras hyperparameter optimization with Talos

2 February 2022 0

Peter

post main image In the two previous posts I showed you my first steps with Keras. I used examples found on the internet and changed the dataset into something trivial, meaning I generate the data myself and know the expected values. But I also told you that I had no idea why parameters like neurons, epochs, batch_size had these values.So what we have is not really a black box. On the outside there are also some switches ...