angle-uparrow-clockwisearrow-counterclockwisearrow-down-uparrow-leftatcalendarcard-listchatcheckenvelopefolderhouseinfo-circlepencilpeoplepersonperson-fillperson-plusphoneplusquestion-circlesearchtagtrashx

Flask, WTForms and AJAX: CSRF protection, before_request and multilanguage

You should always check if CSRF protection is working. With Flask this is not obvious.

29 February 2020 Updated 29 February 2020
post main image
https://unsplash.com/@christineashleydonaldson

I never really checked if CSRF protection was working in my Flask application, this website. Is it enabled by default? From the Flask_WTF extension documentation:

Any view using FlaskForm to process the request is already getting CSRF protection.

And from the text of Miguel Grinberg's post 'Cookie Security for Flask Applications':

If you are handling your web forms with the Flask-WTF extension, you are already protected against CSRF on your forms by default.

It should be enabled, let's check if this is true. On the page with the form I added:

<script>
$(document).ready(function(){
    $('#csrf_token').val('ABC');
});
</script>

Then I refreshed the form. In the debugger I verified that the csrf_token changed to 'ABC'.  Clicking the submit button should give a CSRF error, well I expected a CSRF exception. For me, no exception, which means no CSRF protection. How is this possible? Did it have to do with the fact that I am using DispatcherMiddleWare or is something else wrong?

Yes, something was wrong. Me! And it was caused by TL;DR. There was in fact a CSRF error but not a CSRF exception. I have many forms, mostly in the admin section, and they have been working with CSRF protection all the time without any CSRF error. What is going on?

My testapp:

def create_app():
    fname = 'create_app'
    print(fname + '()')

    app = Flask(__name__)

    app.config['SECRET_KEY'] = 'some-secret-key'

    class MyCSRFForm(FlaskForm):
        submit = SubmitField(_l('Send'))

    @app.route('/csrf_form', methods=['GET', 'POST'])
    def csrf_form():
        fname = 'csrf_form'
        print(fname + '()')

        form = MyCSRFForm()
        if form.validate_on_submit():
            print(fname + ': no validate_on_submit errors, processing form')

        print(fname + ': form.errors = '.format(form.errors))
        for field, errors in form.errors.items():
            print(fname + ': form.errors, field = {}, errors = {}'.format(field, errors))
                
        return render_template('csrf_form.html', form=form)

and the csrf_form.html template:

<html>
<head></head>
<body>

<form method="post">
    {{ form.csrf_token }}
    <p>{{ form.submit() }}</p>
</form>

<script>
elem = document.getElementById('csrf_token');
elem.setAttribute('value', 'ABC'); 
</script>

</body>
</html>

The result:

csrf_form: form.errors, field = csrf_token, errors = ['The CSRF token is invalid.']

There is more information about this in 'Inconsistency with raising CSRFError #381', see the links below. It appears that you can use Flask-WTF CSRF protection in two ways:

  • as a CSRF error message during form validation
  • as a CSRF exception

Both have advantage and disadvantages. I decided to go for the CSRF exception because there is no way you can forget this by accident. To generate CSRF exceptions, I added the suggested code in my __init__.py:

from flask_wtf.csrf import CSRFProtect
csrf = CSRFProtect()
...

def create_app(project_config):
    ...
    # csrf protection
    csrf.init_app(app)

Now when I submit the form I get a CSRF exception:

Bad Request
The CSRF token is invalid.

I changed the jQuery and removed the field name csrf_token:

<script>
$(document).ready(function(){
    $('#csrf_token').attr({name: 'nothing'});
});
</script>

Refresh and submit, the CSRF exception is:

Bad Request
The CSRF token is missing.

And as a final test we remove the script and set the CSRF token expire time to 5 seconds in create_app():

    app.config['WTF_CSRF_TIME_LIMIT'] = 5

The CSRF exception is:

Bad Request
The CSRF token has expired.

Summary of these tests:

  • The http error code is 400 Bad Request
  • It appeared CSRF protection was working by default and can be implemented in two ways, the documentation is not very clear on this
  • I wanted CSRF exceptions and added the suggested extra code
  • It was very easy to generate CSRF exceptions for the conditions:
    - The CSRF token is invalid
    - The CSRF token is missing
    - The CSRF token has expired

Multiple forms and the CSRF token

A blog post page on this site has two types of comment forms: the comment form and the comment reply form, see also a previous post.
When looking at the page with the comment form and comment reply form I noticed that both had the CSRF token value after a page load.

comment form: 
<input id="comment_form-csrf_token" name="comment_form-csrf_token" type="hidden" value="IjM4N2Y3NmEzMGFkNGMwOTY2MzM5OGI0NTVjOGQwNDI3MjY3MmQxOGIi.Xk6ReA.dFbMlM0bClY6BvAVjxZ0GfiFBM4">

comment reply form: 
<input id="comment_reply_form-csrf_token" name="comment_reply_form-csrf_token" type="hidden" value="IjM4N2Y3NmEzMGFkNGMwOTY2MzM5OGI0NTVjOGQwNDI3MjY3MmQxOGIi.Xk6ReA.dFbMlM0bClY6BvAVjxZ0GfiFBM4">

or:

comment_form-csrf_token: IjM4N2Y3NmEzMGFkNGMwOTY2MzM5OGI0NTVjOGQwNDI3MjY3MmQxOGIi.Xk6ReA.dFbMlM0bClY6BvAVjxZ0GfiFBM4

comment_reply_form-csrf_token: IjM4N2Y3NmEzMGFkNGMwOTY2MzM5OGI0NTVjOGQwNDI3MjY3MmQxOGIi.Xk6ReA.dFbMlM0bClY6BvAVjxZ0GfiFBM4

Then I submit a too short message using the comment reply form. The form is send to the server, rendered at the server and send back to the client.
Inspection showed that the CSRF token of the comment reply form changed:

comment_reply_form-csrf_token: IjM4N2Y3NmEzMGFkNGMwOTY2MzM5OGI0NTVjOGQwNDI3MjY3MmQxOGIi.Xk6SUg.6axMGCN2KmQQ4WeAAxYgTg6UdAs

Here the following questions came up:

  • How is the CSRF token constructed?
  • When does the CSRF token expire?
  • Do different forms on a page require different CSRF token?
  • How can the CSRF token of the comment reply form change?
  • What is the X-CSRFToken?

How is the CSRF token constructed?

Time to start reading again. From the OWASP guide:

In general, developers need only generate this token once for the current session. After initial generation of this token, the value is stored in the session and is used for each subsequent request until the session expires.

In Flask-WTF the raw token is in the session and the signed token is in g. Or more accurate, the raw token is always in the session and the signed token is in g after a form has been used. Before instantiating a form:

session['csrf_token']: 387f76a30ad4c09663398b455c8d04272672d18b
g.csrf_token: None

After instantiating a form:

session['csrf_token']: 387f76a30ad4c09663398b455c8d04272672d18b
g.csrf_token: IjM4N2Y3NmEzMGFkNGMwOTY2MzM5OGI0NTVjOGQwNDI3MjY3MmQxOGIi.Xk6vnw.L7uNOptXpBxemh_TIEy3e9Ujllg

The signed token is put in g so that with subsequent requests, for example in case of multiple forms, the same signed token can be used and does not have to be generated again. If you want you can experiment with generating and validating tokens using the functions generate_csrf() and validate_csrf():

    from flask_wtf.csrf import generate_csrf, validate_csrf

    secret_key = current_app.config['SECRET_KEY']
    # default: WTF_CSRF_FIELD_NAME = 'csrf_token'
    token_key = current_app.config['WTF_CSRF_FIELD_NAME']

    csrf_token = generate_csrf(secret_key=secret_key, token_key=token_key)
    current_app.logger.debug(fname + ': csrf_token = {}'.format(csrf_token))

    validated = False
    try:
        validate_csrf(csrf_token, secret_key=secret_key, time_limit=3600, token_key=token_key)
        validated = True
    except:
        pass
    current_app.logger.debug(fname + ': validated = {}'.format(validated))

The above functions do something like a black box. But we can see that the generated token is signed and includes a timestamp.

When does the CSRF token expire?

We already saw that the CSRF token expire time can be controlled with:

    app.config['WTF_CSRF_TIME_LIMIT'] = 5

The default value is 3600, one hour. This time is somewhere in the token. There can be two possible ways how this is implemented:

  • The start time is in the CSRF token
  • The end time is in the CSRF token

This is not important to know unless you want to change the expire time inside your application. The generate_csrf() function does not include a time_limit parameter while the validate_csrf does. This would mean that the start time is in the CSRF token. To verify this I put a page with a form on the screen, then changed the default time_limit to 10 seconds and then submitted the form. As expected, a CSRF exception was raised.

There is another condition when your CSRF token expires and that is when the session expires. The reason is that the CSRF raw token is stored in the session. This means that if your session lives shorter than your CSRF token expire time you can get CSRF errors.

Do different forms on a page require different CSRF token?

Answer: This is more a general question. The answer is no. We can use the same CSRF token for all forms on a page.

How can the CSRF token of the comment reply form change?

Answer: The token does not change but the signed token changes, because there also is a timestamp in the signed token. If you look at the signed tokens above you notice that the first part is identical.

The X-CSRFToken header parameter

You may have read about the X-CSRFToken. The X-CSRFToken is a setting in the header. Usually we do not need this. If you have a CSRF token parameter in your AJAX POST then this will be used. If not then can set this in the header of the POST using jQuery:

var csrf_token = ...

    ...
    $.ajax({
        beforeSend: function(xhr, settings){
            if(!/^(GET|HEAD|OPTIONS|TRACE)$/i.test(settings.type) && !this.crossDomain){
                xhr.setRequestHeader("X-CSRFToken", csrf_token);
            }
        },

It depends on your implementation if you send forms with a CSRF token parameter or if you set the X-CSRFToken header parameter. The documentation states you can only use this in 'CSRF exception mode' (adding the extra code). I did not check this.

Handling CSRF exceptions and before_request

I already handle HTTP error exceptions like 401 (Unauthorized), 403 (Forbidden), 404 (Not Found), 405 (Method Not Allowed), 500 (Internal Server Error) but how do we handle the CSRF exception? The CSRF error propagates to a 400 (Bad Request).

I created nice bells and whistles error pages showing the exceptions. I did this by relying on the fact that that before_request was called.
In before_request I process the incoming request and among other things, set the selected language accordingly.

Unfortunately by default before_request is NOT called on a CSRF exception. But there is a way around this. We can set:

    app.config['WTF_CSRF_CHECK_DEFAULT'] = False

The result will be that before_request is called. Then in before_request, after some essential processing, we can call:

    csrf.protect()

This will raise a CSFR exception, if there is one, and call the error handler accordingly.

Multi-language CSRF exception messages

I use Flask-Babel and validation messages are translated into the selected language. The CSRF exception messages did not translate and checking csrf.py it appeared that these messages are hard-coded in English(?). For the moment I created a dictionary to handle the translation:

    csrf_error_message2translated_error_messages = {
        ...
        'The CSRF token has expired.': _('The CSRF token has expired.'),
        ...
    }
    if error.description in error_description2error_messages:
        error_message = error_description2error_messages[error.description]
    else:
        error_message = error.description

Responding to exceptions, including CSRF, on AJAX requests

With AJAX we cannot show our nice HTML error page on an exception. On a request we must return an error code or HTML that the client understands. It comes down to returning the exception as a JSON encoded error message to the client. In Flask we already have our error handlers.
But how do we know that the request was coming from an AJAX call? I am sending the form with encoding application/x-www-form-urlencoded:

    $.ajax({
        data: $('#' + form_id).serialize() + '&' + encodeURI( $(submit_button).attr('name') ) + '=' + encodeURI( $(submit_button).val() ),
        type: 'POST',
        url: url_comment_new,
        contentType: 'application/x-www-form-urlencoded; charset=UTF-8', 
        dataType: 'json'
    })

One way is to look at the received Accept header.

    current_app.logger.error(fname + ': request.accept_mimetypes = {}'.format(request.accept_mimetypes))
    # text/javascript,application/json,*/*;q=0.01

From the document 'List of default Accept values':

Note that all browsers add the */* MIME Type to cover all cases. This is typically used for requests initiated via the address bar of a browser, or via an HTML <a> element.

This means that both 'request.accept_mimetypes.accept_json' and 'request.accept_mimetypes.accept_html' are True. Not very useful. The accept_mimetypes are listed in order of preference. We could scan the list and decide to return JSON when encountering 'application/json' before 'text/html'. But I do not know exactly which browsers send which accept_mimetypes, meaning that I am not 100% sure at this moment that the 'without AJAX' error pages will not accidently be shown as JSON.

A much safer way is to add a parameter to AJAX requests and check in the error handler if this parameter is present. If it is present then we return a JSON error message, if it is not we output the standard error page. I do not like this but it works and some time will dig deeper into this.

Summary

You should always check if CSRF protection is working. I did not, and that was the start of very much reading. It appeared that are two ways to deal with CSRF protection, CSRF form error and CSRF exception. I choose the second one. For me, handling CSRF protection for this multilanguage site was not standard, the main reason being that before_request is not called by default on a CSRF exception. In before_request I check and set the language among other things so this really must run. Fortunately we can delay the CSRF protection and call it explicitly in before_request after essential processing using csrf.protect(). We already have our exception error pages but with AJAX requests we must return JSON error messages. Detecting if the request comes from an AJAX request is complex so I used an extra parameter in the AJAX requests.

Links / credits

Cookie Security for Flask Applications
https://blog.miguelgrinberg.com/post/cookie-security-for-flask-applications

Cross-Site Request Forgery (CSRF) Prevention Cheat Sheet
https://owasp.org/www-project-cheat-sheets/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#Prevention_Measures_That_Do_NOT_Work

Flask-WTF CSRF Protection
https://flask-wtf.readthedocs.io/en/latest/csrf.html

Generating CSRF tokens for multiple forms on a single page
https://stackoverflow.com/questions/14715250/generating-csrf-tokens-for-multiple-forms-on-a-single-page

How to use Flask-WTForms CSRF protection with AJAX?
https://stackoverflow.com/questions/31888316/how-to-use-flask-wtforms-csrf-protection-with-ajax

Inconsistency with raising CSRFError #381
https://github.com/lepture/flask-wtf/issues/381

List of default Accept values
https://developer.mozilla.org/en-US/docs/Web/HTTP/Content_negotiation/List_of_default_Accept_values

Posting a WTForm via AJAX with Flask
https://medium.com/@doobeh/posting-a-wtform-via-ajax-with-flask-b977782edeee

Unacceptable Browser HTTP Accept Headers (Yes, You Safari and Internet Explorer)
https://www.newmediacampaigns.com/blog/browser-rest-http-accept-headers

Leave a comment

Comment anonymously or log in to comment.

Comments (1)

Leave a reply

Reply anonymously or log in to reply.

avatar

Thanks for placing this post.
I have a problem with my webapp that raises CSRF errors, which I asked in
https://stackoverflow.com/questions/65985400/getting-csrf-errors-from-put-fetch-request-that-does-not-involve-flask-forms
Can you please explain why I'm getting CSRF error in the first place, and how to fix the problem?
Thanks again.
Avner