angle-uparrow-clockwisearrow-counterclockwisearrow-down-uparrow-leftatcalendarcard-listchatcheckenvelopefolderhouseinfo-circlepencilpeoplepersonperson-fillperson-plusphoneplusquestion-circlesearchtagtrashx

Welcome

This website is a multi-language CMS / Blog I build with Python, the framework Flask and SQLAlchemy. It is deployed on my ISPConfig3 (Nginx + Mariadb) server using Docker which is also used for development. Read more about this website.

Coronavirus

I investigated the deaths in The Netherlands in 2020 and looking at the data, I can only wonder why we have never had, and still do not have, a serious discussion about reopening the economy for people under 65 or, for example, people under 50. Read the findings and conclusions.

Recent

Most viewed

Tags

AiohttpAJAXAlembicAlpineAPIApplication settingsAsyncBabelBeautifulSoupBootstrapCachingCaptchaCeleryColorsCompileCronCryptographyCSRF protectionCythonDatabaseDecoratorDeep LearningDeepLDispatcherMiddleWareDistributionDNSDockerDocker SwarmDocker-composeEmailExceptionsFastAPIFlaskGunicornIconsIMAPInternetISPConfigJavascriptJinja2KubernetesLog fileLoggingMachine LearningMariaDBMessage brokerMicrok8sMultilanguageMultiprocessingPentestingPikaPostfixPostgreSQLProfilingPyInstallerpyOpenSSLPytestRabbitMQRedisrqliteSchedulingScrapingSecuritySecurity testingSeleniumSlackSQLAlchemyTestingThreadsTimezoneToastsUbuntu TouchUUIDWeb automationWerkzeugWheelsWTFormsYouTube

Using Ingress to access RabbitMQ on a Microk8s cluster

20 November 2024 0

Peter

post main image Some time ago I made a post about moving part of an application containing RabbitMQ running on Docker Swarm to Microk8s. We used NodePort to access the Kubernetes cluster RabbitMQ on the host. Recently I revisted this and decided to use the Microk8s Ingress Controller to access RabbitMQ.In this post, we assume the RabbitMQ pod is present and only focus on getting the Ingress Controller up and running. Below, ...

Simple video galery with Flask, Jinja, Bootstrap and JQuery

28 October 2024 0

Peter

post main image We have a number of cameras that generate short clips when something moves in front of the camera. All the clips, videos, end up on one system. We use standard programs to view the videos. But now we want to share these videos with others on our local network, and we don't want to copy the video files. An obvious solution is to select and install something like a video gallery server. But, hey, this ...

Basic job scheduling with APScheduler

19 October 2024 0

Peter

post main image You have created a Python program, for example, to back up a database, and now you want to run it every hour, or every day. There are many solutions to choose from, here we use APScheduler, 'Advanced Python Scheduler', a job scheduler that meets many requirements. For our task, this package may seem like overkill, but let's use it here and once we understand the basics, we may want to use it for more ...

A database switch with HAProxy and the HAProxy Runtime API

13 August 2024 0

Peter

post main image One of my projects needed a high availability database (don't we all want this ...).  This means (asynchronous) replication, and a multi-node setup. Several solutions exist, will write about this in another post. In these scenarios we have multiple replicas of the main database, and when a problem occurs, we switch from one database to another. To do this switching of databases without touching ...

Docker Swarm rolling updates

7 July 2024 0

Peter

post main image Some time ago I wrote that it would be best to move to a Kubernetes variant and now this post is about Docker Swarm. Yes, I still use Docker Swarm because I have a project that uses it. I recently moved development from Docker to Docker Swarm, mainly because with Docker Swarm you learn the basics of container orchestration, so why not learn this during development. In this post, we'll look at rolling ...

Hiding database UUID primary keys of your web application

29 March 2024 0

Peter

post main image When you create a web application, you must be very careful not to expose too much information. If you use (auto-increment) Integer IDs in your database, then you are probably exposing too much already. Some examples. An Integer user_id makes it easy to guess how many new registrations you receive every day. An Integer order_id makes it easy to guess how many orders you receive every day. In addition, ...

Don't Repeat Yourself (DRY) with Jinja2

20 February 2024 0

Peter

post main image I was trying some things with Jinja2, created a small Flask app, and thought why not share this. What I wanted to achieve was to put all page names in one template file. As with Python, when writing a lot of code, you must be careful not to repeat yourself with Jinja2. Before you know it, you end up with many template files containing the same types of information. And when you want to change something, ...

SQLAlchemy, PostgreSQL, maximum number of rows per user

5 February 2024 0

Peter

post main image You have a multi-user application using SQLAlchemy and PostgreSQL and want to limit the number of rows per user of a certain table. For example, every user can have a maximum of five posts. You need an operation like: Lock the table Count the number of posts of the user If the number less than five: Add new post Unlock the table Else: Unlock the table Generate exception ...

Show the values in SQLAlchemy dynamic filters

18 January 2024 0

Peter

post main image When using SQLAlchemy, I often use dynamic filters in my queries. This means that I start with a list with some conditions and add more conditions that depend on other variables.Here is a query with a static filter: # query with static filter product_colors = ['white'] stmt = sa.select(Product).\ where(sa.and( Product.category.in_(my_categories), Product.color.in_(product_colors), ...

Secure data transfer with Public Key encryption and pyNaCl

2 December 2023 0

Peter

post main image This is a short post about transferring data safely between two persons. For this we use the Python pyNaCl package to generate private and public keys and to encrypt and decrypt the data. I also added the Python keyring package to store the private_key and public_key. Not really that difficult. I needed a basic class to do this and here I share it. Maybe you find it useful. As always I am developing ...

rqlite: a high-availability and distributed SQLite alternative

17 October 2023 1

Peter

post main image In a project I am using a SQLite database. The data is not critical, it can be reloaded at any time. Still, I do not want part of the application to become unresponsive when the SQLite database is temporarily unavailable. I was looking for a fast, more or less fault-tolerant database, and also distributed, so I can replicate some reader modules. There are a few solutions that came up when searching ...

Should I migrate my Docker Swarm to Kubernetes?

15 September 2023 0

Peter

post main image When you read posts on the internet saying that Docker Swarm is dead, you get scared. I have a Docker Swarm running and I like it, it's easy when you already use Docker. What are the alternatives? We read the whole time that there is only one thing to do and that is to migrate to Kubernetes and forget everything else. I was half-way migrating from Docker to Docker Swarm, and wanted to know if I should ...

Get a list of YouTube videos of a person

7 September 2023 0

Peter

post main image A few days ago I got the question: Can you download all the public YouTube videos of a person, that were uploaded between 2020 and today. The total number of videos was about two hundred. And no, I could not get access to this person's YouTube account. In this post, I use the YouTube API to download the required metadata from the videos, one item per video. I looked in PyPI, but could not find a suitable ...

From Docker-Compose to Docker Swarm: Configs

29 August 2023 0

Peter

post main image You have an application that consists of a number of Docker-Compose projects, and are using Docker-Compose to build, start and deploy. Now you want to go one step further and move some of the Docker-Compose projects to another server. The most obvious choice to do this, well to try first, is Docker Swarm. You learn a few more Docker commands and you're done. Is it really that easy? Spoiler alert. ...

Docker-Compose projects with identical service names

25 August 2023 0

Peter

post main image If we have identical Docker-Compose projects with identical service names, connected by a Docker network, we must make sure that we access the proper service. Over a Docker network there are two ways we can access a service: By service name By container name I have multiple Docker-Compose projects that are almost identical, each project is in its own directory and has its own environment. I ...

X web automation and scraping with Selenium

11 August 2023 0

Peter

post main image When you want to scrape data from the Web, you have to know what you're doing. You don't want to overload a target server with requests. If you do this from a single location, an IP address, you could get a (temporary) ban. If you want to scrape big, consider using a dedicated service such as ZenRows, ScrapFly, WebScrapingAPI, ScrapingAnt, etc. They distribute your requests across a lot of systems, ...

Aiohttp with custom DNS servers, Unbound and Docker

13 July 2023 0

Peter

post main image Using aiohttp looks so easy, but it is not. It's confusing. The 'Client Quickstart' documentation begins with the following: Note Don’t create a session per request. Most likely you need a session per application which performs all requests together. More complex cases may require a session per site, e.g. one for Github and other one for Facebook APIs. Anyway making a session for every ...

Return only the values of a list of records from FastAPI

6 July 2023 0

Peter

post main image In Python, everything is a class, which means that model data is similar to a dictionary. But dictionaries have keys. And when you return a list of many dictionaries from FastAPI, the size of the data, keys and values, is usually much more than twice the size of the values. Larger size and more time means that our application is not very efficient, slower than necessary. It also means it consumes more ...

Prevent IP address spoofing using Reverse path filtering

15 June 2023 0

Peter

post main image This post is about system administration and has nothing to do with Python. Then why post this here? Because I believe there many like me who run one or more web servers and sometime run into these problems. In the previous post I wrote that my ISPConfig Debian server was subject to port scanning, etc. and that it appeared that 95% of all requests came from China unless ... these IP addresses were ...

Collect and block IP addresses with ipset and Python

21 May 2023 0

Peter

post main image If you have a server connected to the Internet, you've probably seen this in your log files: lots of illegal external requests trying to access your services. I maintain a Debian server, and use Fail2Ban for intrusion prevention. Standard practice, install, configure and forget. Since the server was getting pulled down at certain times, I decided to take a closer look. I'm mostly into programming ...