- Using UUIDs instead of Integer Autoincrement Primary Keys with SQLAlchemy and MariaDb
- Using Python's pyOpenSSL to verify SSL certificates downloaded from a host
- Using PyInstaller and Cython to create a Python executable
- Connect to a service on a Docker host from a Docker container
- SQLAlchemy: Using Cascade Deletes to delete related objects
- Flask RESTful API request parameter validation with Marshmallow schemas
7 September 2023 0
A few days ago I got the question: Can you download all the public YouTube videos of a person, that were uploaded between 2020 and today. The total number of videos was about two hundred. And no, I could not get access to this person's YouTube account. In this post, I use the YouTube API to download the required metadata from the videos, one item per video. I looked in PyPI, but could not find a suitable ...
29 August 2023 0
You have an application that consists of a number of Docker-Compose projects, and are using Docker-Compose to build, start and deploy. Now you want to go one step further and move some of the Docker-Compose projects to another server. The most obvious choice to do this, well to try first, is Docker Swarm. You learn a few more Docker commands and you're done. Is it really that easy? Spoiler alert. ...
25 August 2023 0
If we have identical Docker-Compose projects with identical service names, connected by a Docker network, we must make sure that we access the proper service. Over a Docker network there are two ways we can access a service: By service name By container name I have multiple Docker-Compose projects that are almost identical, each project is in its own directory and has its own environment. I ...
11 August 2023 0
When you want to scrape data from the Web, you have to know what you're doing. You don't want to overload a target server with requests. If you do this from a single location, an IP address, you could get a (temporary) ban. If you want to scrape big, consider using a dedicated service such as ZenRows, ScrapFly, WebScrapingAPI, ScrapingAnt, etc. They distribute your requests across a lot of systems, ...
13 July 2023 0
Using aiohttp looks so easy, but it is not. It's confusing. The 'Client Quickstart' documentation begins with the following: Note Don’t create a session per request. Most likely you need a session per application which performs all requests together. More complex cases may require a session per site, e.g. one for Github and other one for Facebook APIs. Anyway making a session for every ...
6 July 2023 0
In Python, everything is a class, which means that model data is similar to a dictionary. But dictionaries have keys. And when you return a list of many dictionaries from FastAPI, the size of the data, keys and values, is usually much more than twice the size of the values. Larger size and more time means that our application is not very efficient, slower than necessary. It also means it consumes more ...
15 June 2023 0
This post is about system administration and has nothing to do with Python. Then why post this here? Because I believe there many like me who run one or more web servers and sometime run into these problems. In the previous post I wrote that my ISPConfig Debian server was subject to port scanning, etc. and that it appeared that 95% of all requests came from China unless ... these IP addresses were ...
21 May 2023 0
If you have a server connected to the Internet, you've probably seen this in your log files: lots of illegal external requests trying to access your services. I maintain a Debian server, and use Fail2Ban for intrusion prevention. Standard practice, install, configure and forget. Since the server was getting pulled down at certain times, I decided to take a closer look. I'm mostly into programming ...
2 May 2023 0
For a project I was using AIOHTTP to check the responses of many remote websites, URLs. The URLs were coming from a list. This list can contain duplicates. Everything fine until I noticed that some responses also had status code: HTTP-429 'Too Many Requests'. Whatever the reason, overload, security, we want to behave friendly and do not want to call identical URLs again, at least for a minimum time. ...
18 April 2023 0
When using Docker, your application typically consists of several Docker containers. Often, you want to run scripts inside these containers at certain moments, for example, every five minutes, once an hour, once a day. This is where the job scheduler Cron comes in, and there are several options on how to do this. In this post I create a separate Cron container, and use the Docker Exec command to execute ...
10 April 2023
In the past I wrote some code for a (text-only) Captcha for a Flask website. This is an update. Here I use a Pypi package to generate the image. Besides that I also added a Captcha image refresh button.You can try the code below, it is a (newsletter) subscription form. Although this is a Captcha solution for a Flask website, it can be converted into a Captcha server. Note that I use SQLite here for ...
30 March 2023 0
I was working on a project with SQLAlchemy and PostgreSQL. For a few tables, I wanted to limit the number of rows per user, and did this by adding a PostgreSQL check function and trigger. Manual testing every thing appeared to be working fine but what if a user would start multiple processes and add rows at exactly the same time? I added the 'pg_advisory_xact_lock' but will this really work? Did I ...
23 March 2023 0
For a project I was already sending messages by email, but now I also wanted to send messages to Slack. Of course, we are using the Python Slack SDK. The documentation can be found on the page: Python Slack SDK - Web Client. In this post I create a simple SlackAPI class with its own SlackError exception class. Create and configure a new Slack App We will be sending our messages to a Slack Channel. ...
9 March 2023 0
This is a short post about backup of a Dockerized PostgreSQL database. To access the database, we typically run a Bash script on the host, with commands like: docker exec -t <container> bash -c '<command>' In this post we replace our Bash script with a Python script. Why? Because we know Python and programming in Bash can be time-consuming. While we can use 'subprocess' to run all system ...
21 February 2023 0
Many times applications must send messages to a remote system. In a perfect world we only have to deal with the Happy Flow: There are no bad things happening like exceptions, errors. Unfortunately, the world is not perfect. Besides programming errors, connections can fail, database systems can fail, remote systems can fail. A simple question - can you write some code to send messages to a remote system ...
11 January 2023 0
This is a short post. All the time we hear these politicians talking but most of the time they mean the opposite. For example, if a politician says that he will lower the taxes, the taxes will go up. If a politician says he did not a relationship with that woman, then ... Etc. So I thought, why not make a Politician Translator in Python? In this post, I start with the results. The code is at the end. The ...
8 January 2023 0
This post is about using RabbitMQ in your Python application. If you are using Rabbit already, you will probably find nothing useful in this post. Why a post about RabbitMQ? Because I have an application using it.It is running for a year, and I thought to share my experiences. In this post, we transform a monolithic application into services decoupled by queues. Also, I will talk about services instead ...
19 December 2022 0
In a Flask project, I needed to run a background job, more specifically a command running in a (Linux) terminal, and show its output, stdout and sterr, in real time in a browser window. You can find some solutions on the internet and this is just another one. I am also using some code I found on the web, see links below. This solution is using: multiprocessing, to start a new process from our Flask ...
11 December 2022 0
The problem: You finally have a Docker application consisting of many (micro) services (containers) and want to monitor all these containers for errors. The services are mostly Python scripts that use the standard Python logging module and print messages to stdout (and stderr): ... DEBUG ... ... INFO ... ... ERROR ... Using the default Docker json-file logging driver, these messages end up in ...
5 December 2022 0
When you develop a software application, probably the first thing you will set up is logging. First only to the console, but soon you will add log files. Some years ago I started using Docker for development and production. I did not make much changes when moving to Docker. The application log files are still in a log directory on a Docker Volume. This means that the log files are part of the application ...