Test failures in tutorial with python3.12

[sblondon]

The tests runned with pytest works with python3.11 and not in 3.12:

How to reproduce:

git clone [email protected]:pallets/flask.git
cd flask/examples/tutorial
python3.11 -m venv venv3.11
python3.12 -m venv venv3.12
# install dependencies
# setup db

In both cases, the webservice runs properly with ./venv3.1x/bin/flask --app flaskr run --debug.

However, running the tests show different results. In venv3.11, the 24 tests are green. In venv3.12 virtualenv, there is 6 failures (and 18 green). The failures are due to a DeprecationWarning which becomes an error:

DeprecationWarning: The default timestamp converter is deprecated as of Python 3.12; see the sqlite3 documentation for suggested replacement recipes

The full output is provided at the end of the bug report.

The DeprecationWarning is documented in the 3.12 release: 'default adapters and converters are now deprecated. Instead, use the Adapter and converter recipes and tailor them to your needs.' Copy-pasting blindly the recipes in tests/conftest.py, flaskr/__init__.py and flaskr/db.py does not fix the errors.

The errors can be fixed by adding thoses lines in conftest.py:

def convert_timestamp(val):
    """Convert Unix epoch timestamp to datetime.datetime object."""
    return datetime.datetime.strptime(val.decode("utf-8"), "%Y-%m-%d %H:%M:%S").replace(tzinfo=datetime.timezone.utc)

sqlite3.register_converter("timestamp", convert_timestamp)

It's probably not the best fix but it's a start.

Pytest output:

(venv3.12) $ ./venv3.12/bin/pytest
========================================= test session starts ==========================================
platform linux -- Python 3.12.4, pytest-8.3.2, pluggy-1.5.0
rootdir: /home/stephane/src/flasktuto/flask/examples/tutorial
configfile: pyproject.toml
testpaths: tests
collected 24 items                                                                                     

tests/test_auth.py ....F...                                                                      [ 33%]
tests/test_blog.py F...F...F.FF                                                                  [ 83%]
tests/test_db.py ..                                                                              [ 91%]
tests/test_factory.py ..                                                                         [100%]

=============================================== FAILURES ===============================================
______________________________________________ test_login ______________________________________________

client = <FlaskClient <Flask 'flaskr'>>, auth = <conftest.AuthActions object at 0x7f297abc6900>

    def test_login(client, auth):
        # test that viewing the page renders without template errors
        assert client.get("/auth/login").status_code == 200
    
        # test that successful login redirects to the index page
        response = auth.login()
        assert response.headers["Location"] == "/"
    
        # login request set the user_id in the session
        # check that the user is loaded from the session
        with client:
>           client.get("/")

tests/test_auth.py:50: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:1162: in get
    return self.open(*args, **kw)
venv3.12/lib/python3.12/site-packages/flask/testing.py:235: in open
    response = super().open(
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:1116: in open
    response_parts = self.run_wsgi_app(request.environ, buffered=buffered)
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:988: in run_wsgi_app
    rv = run_wsgi_app(self.application, environ, buffered=buffered)
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:1264: in run_wsgi_app
    app_rv = app(environ, start_response)
venv3.12/lib/python3.12/site-packages/flask/app.py:1498: in __call__
    return self.wsgi_app(environ, start_response)
venv3.12/lib/python3.12/site-packages/flask/app.py:1476: in wsgi_app
    response = self.handle_exception(e)
venv3.12/lib/python3.12/site-packages/flask/app.py:1473: in wsgi_app
    response = self.full_dispatch_request()
venv3.12/lib/python3.12/site-packages/flask/app.py:882: in full_dispatch_request
    rv = self.handle_user_exception(e)
venv3.12/lib/python3.12/site-packages/flask/app.py:880: in full_dispatch_request
    rv = self.dispatch_request()
venv3.12/lib/python3.12/site-packages/flask/app.py:865: in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
flaskr/blog.py:24: in index
    ).fetchall()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

val = b'2018-01-01 00:00:00'

    def convert_timestamp(val):
>       warn(msg.format(what="timestamp converter"), DeprecationWarning, stacklevel=2)
E       DeprecationWarning: The default timestamp converter is deprecated as of Python 3.12; see the sqlite3 documentation for suggested replacement recipes

/usr/lib/python3.12/sqlite3/dbapi2.py:76: DeprecationWarning
______________________________________________ test_index ______________________________________________

client = <FlaskClient <Flask 'flaskr'>>, auth = <conftest.AuthActions object at 0x7f297ad26390>

    def test_index(client, auth):
>       response = client.get("/")

tests/test_blog.py:7: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:1162: in get
    return self.open(*args, **kw)
venv3.12/lib/python3.12/site-packages/flask/testing.py:235: in open
    response = super().open(
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:1116: in open
    response_parts = self.run_wsgi_app(request.environ, buffered=buffered)
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:988: in run_wsgi_app
    rv = run_wsgi_app(self.application, environ, buffered=buffered)
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:1264: in run_wsgi_app
    app_rv = app(environ, start_response)
venv3.12/lib/python3.12/site-packages/flask/app.py:1498: in __call__
    return self.wsgi_app(environ, start_response)
venv3.12/lib/python3.12/site-packages/flask/app.py:1476: in wsgi_app
    response = self.handle_exception(e)
venv3.12/lib/python3.12/site-packages/flask/app.py:1473: in wsgi_app
    response = self.full_dispatch_request()
venv3.12/lib/python3.12/site-packages/flask/app.py:882: in full_dispatch_request
    rv = self.handle_user_exception(e)
venv3.12/lib/python3.12/site-packages/flask/app.py:880: in full_dispatch_request
    rv = self.dispatch_request()
venv3.12/lib/python3.12/site-packages/flask/app.py:865: in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
flaskr/blog.py:24: in index
    ).fetchall()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

val = b'2018-01-01 00:00:00'

    def convert_timestamp(val):
>       warn(msg.format(what="timestamp converter"), DeprecationWarning, stacklevel=2)
E       DeprecationWarning: The default timestamp converter is deprecated as of Python 3.12; see the sqlite3 documentation for suggested replacement recipes

/usr/lib/python3.12/sqlite3/dbapi2.py:76: DeprecationWarning
_________________________________________ test_author_required _________________________________________

app = <Flask 'flaskr'>, client = <FlaskClient <Flask 'flaskr'>>
auth = <conftest.AuthActions object at 0x7f297ad3e210>

    def test_author_required(app, client, auth):
        # change the post author to another user
        with app.app_context():
            db = get_db()
            db.execute("UPDATE post SET author_id = 2 WHERE id = 1")
            db.commit()
    
        auth.login()
        # current user can't modify other user's post
>       assert client.post("/1/update").status_code == 403

tests/test_blog.py:34: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:1167: in post
    return self.open(*args, **kw)
venv3.12/lib/python3.12/site-packages/flask/testing.py:235: in open
    response = super().open(
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:1116: in open
    response_parts = self.run_wsgi_app(request.environ, buffered=buffered)
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:988: in run_wsgi_app
    rv = run_wsgi_app(self.application, environ, buffered=buffered)
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:1264: in run_wsgi_app
    app_rv = app(environ, start_response)
venv3.12/lib/python3.12/site-packages/flask/app.py:1498: in __call__
    return self.wsgi_app(environ, start_response)
venv3.12/lib/python3.12/site-packages/flask/app.py:1476: in wsgi_app
    response = self.handle_exception(e)
venv3.12/lib/python3.12/site-packages/flask/app.py:1473: in wsgi_app
    response = self.full_dispatch_request()
venv3.12/lib/python3.12/site-packages/flask/app.py:882: in full_dispatch_request
    rv = self.handle_user_exception(e)
venv3.12/lib/python3.12/site-packages/flask/app.py:880: in full_dispatch_request
    rv = self.dispatch_request()
venv3.12/lib/python3.12/site-packages/flask/app.py:865: in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
flaskr/auth.py:27: in wrapped_view
    return view(**kwargs)
flaskr/blog.py:90: in update
    post = get_post(id)
flaskr/blog.py:48: in get_post
    .fetchone()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

val = b'2018-01-01 00:00:00'

    def convert_timestamp(val):
>       warn(msg.format(what="timestamp converter"), DeprecationWarning, stacklevel=2)
E       DeprecationWarning: The default timestamp converter is deprecated as of Python 3.12; see the sqlite3 documentation for suggested replacement recipes

/usr/lib/python3.12/sqlite3/dbapi2.py:76: DeprecationWarning
_____________________________________________ test_update ______________________________________________

client = <FlaskClient <Flask 'flaskr'>>, auth = <conftest.AuthActions object at 0x7f297abc7920>
app = <Flask 'flaskr'>

    def test_update(client, auth, app):
        auth.login()
>       assert client.get("/1/update").status_code == 200

tests/test_blog.py:59: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:1162: in get
    return self.open(*args, **kw)
venv3.12/lib/python3.12/site-packages/flask/testing.py:235: in open
    response = super().open(
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:1116: in open
    response_parts = self.run_wsgi_app(request.environ, buffered=buffered)
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:988: in run_wsgi_app
    rv = run_wsgi_app(self.application, environ, buffered=buffered)
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:1264: in run_wsgi_app
    app_rv = app(environ, start_response)
venv3.12/lib/python3.12/site-packages/flask/app.py:1498: in __call__
    return self.wsgi_app(environ, start_response)
venv3.12/lib/python3.12/site-packages/flask/app.py:1476: in wsgi_app
    response = self.handle_exception(e)
venv3.12/lib/python3.12/site-packages/flask/app.py:1473: in wsgi_app
    response = self.full_dispatch_request()
venv3.12/lib/python3.12/site-packages/flask/app.py:882: in full_dispatch_request
    rv = self.handle_user_exception(e)
venv3.12/lib/python3.12/site-packages/flask/app.py:880: in full_dispatch_request
    rv = self.dispatch_request()
venv3.12/lib/python3.12/site-packages/flask/app.py:865: in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
flaskr/auth.py:27: in wrapped_view
    return view(**kwargs)
flaskr/blog.py:90: in update
    post = get_post(id)
flaskr/blog.py:48: in get_post
    .fetchone()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

val = b'2018-01-01 00:00:00'

    def convert_timestamp(val):
>       warn(msg.format(what="timestamp converter"), DeprecationWarning, stacklevel=2)
E       DeprecationWarning: The default timestamp converter is deprecated as of Python 3.12; see the sqlite3 documentation for suggested replacement recipes

/usr/lib/python3.12/sqlite3/dbapi2.py:76: DeprecationWarning
________________________________ test_create_update_validate[/1/update] ________________________________

client = <FlaskClient <Flask 'flaskr'>>, auth = <conftest.AuthActions object at 0x7f297ace5160>
path = '/1/update'

    @pytest.mark.parametrize("path", ("/create", "/1/update"))
    def test_create_update_validate(client, auth, path):
        auth.login()
>       response = client.post(path, data={"title": "", "body": ""})

tests/test_blog.py:71: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:1167: in post
    return self.open(*args, **kw)
venv3.12/lib/python3.12/site-packages/flask/testing.py:235: in open
    response = super().open(
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:1116: in open
    response_parts = self.run_wsgi_app(request.environ, buffered=buffered)
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:988: in run_wsgi_app
    rv = run_wsgi_app(self.application, environ, buffered=buffered)
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:1264: in run_wsgi_app
    app_rv = app(environ, start_response)
venv3.12/lib/python3.12/site-packages/flask/app.py:1498: in __call__
    return self.wsgi_app(environ, start_response)
venv3.12/lib/python3.12/site-packages/flask/app.py:1476: in wsgi_app
    response = self.handle_exception(e)
venv3.12/lib/python3.12/site-packages/flask/app.py:1473: in wsgi_app
    response = self.full_dispatch_request()
venv3.12/lib/python3.12/site-packages/flask/app.py:882: in full_dispatch_request
    rv = self.handle_user_exception(e)
venv3.12/lib/python3.12/site-packages/flask/app.py:880: in full_dispatch_request
    rv = self.dispatch_request()
venv3.12/lib/python3.12/site-packages/flask/app.py:865: in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
flaskr/auth.py:27: in wrapped_view
    return view(**kwargs)
flaskr/blog.py:90: in update
    post = get_post(id)
flaskr/blog.py:48: in get_post
    .fetchone()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

val = b'2018-01-01 00:00:00'

    def convert_timestamp(val):
>       warn(msg.format(what="timestamp converter"), DeprecationWarning, stacklevel=2)
E       DeprecationWarning: The default timestamp converter is deprecated as of Python 3.12; see the sqlite3 documentation for suggested replacement recipes

/usr/lib/python3.12/sqlite3/dbapi2.py:76: DeprecationWarning
_____________________________________________ test_delete ______________________________________________

client = <FlaskClient <Flask 'flaskr'>>, auth = <conftest.AuthActions object at 0x7f297ada5610>
app = <Flask 'flaskr'>

    def test_delete(client, auth, app):
        auth.login()
>       response = client.post("/1/delete")

tests/test_blog.py:77: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:1167: in post
    return self.open(*args, **kw)
venv3.12/lib/python3.12/site-packages/flask/testing.py:235: in open
    response = super().open(
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:1116: in open
    response_parts = self.run_wsgi_app(request.environ, buffered=buffered)
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:988: in run_wsgi_app
    rv = run_wsgi_app(self.application, environ, buffered=buffered)
venv3.12/lib/python3.12/site-packages/werkzeug/test.py:1264: in run_wsgi_app
    app_rv = app(environ, start_response)
venv3.12/lib/python3.12/site-packages/flask/app.py:1498: in __call__
    return self.wsgi_app(environ, start_response)
venv3.12/lib/python3.12/site-packages/flask/app.py:1476: in wsgi_app
    response = self.handle_exception(e)
venv3.12/lib/python3.12/site-packages/flask/app.py:1473: in wsgi_app
    response = self.full_dispatch_request()
venv3.12/lib/python3.12/site-packages/flask/app.py:882: in full_dispatch_request
    rv = self.handle_user_exception(e)
venv3.12/lib/python3.12/site-packages/flask/app.py:880: in full_dispatch_request
    rv = self.dispatch_request()
venv3.12/lib/python3.12/site-packages/flask/app.py:865: in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
flaskr/auth.py:27: in wrapped_view
    return view(**kwargs)
flaskr/blog.py:121: in delete
    get_post(id)
flaskr/blog.py:48: in get_post
    .fetchone()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

val = b'2018-01-01 00:00:00'

    def convert_timestamp(val):
>       warn(msg.format(what="timestamp converter"), DeprecationWarning, stacklevel=2)
E       DeprecationWarning: The default timestamp converter is deprecated as of Python 3.12; see the sqlite3 documentation for suggested replacement recipes

/usr/lib/python3.12/sqlite3/dbapi2.py:76: DeprecationWarning
======================================= short test summary info ========================================
FAILED tests/test_auth.py::test_login - DeprecationWarning: The default timestamp converter is deprecated as of Python 3.12; see the sqlite...
FAILED tests/test_blog.py::test_index - DeprecationWarning: The default timestamp converter is deprecated as of Python 3.12; see the sqlite...
FAILED tests/test_blog.py::test_author_required - DeprecationWarning: The default timestamp converter is deprecated as of Python 3.12; see the sqlite...
FAILED tests/test_blog.py::test_update - DeprecationWarning: The default timestamp converter is deprecated as of Python 3.12; see the sqlite...
FAILED tests/test_blog.py::test_create_update_validate[/1/update] - DeprecationWarning: The default timestamp converter is deprecated as of Python 3.12; see the sqlite...
FAILED tests/test_blog.py::test_delete - DeprecationWarning: The default timestamp converter is deprecated as of Python 3.12; see the sqlite...
===================================== 6 failed, 18 passed in 8.05s =====================================

``

[sblondon]

For info, the tests with python3.11 are still green with the dirty patch.

[sblondon]

Trying to write aware datetime by converting the datetime inside the SQL request fails. For example, in blog.index() function:

    posts = db.execute(
        "SELECT p.id, title, body, datetime(p.created, 'localtime'), author_id, username"
        " FROM post p JOIN user u ON p.author_id = u.id"
        " ORDER BY datetime(p.created, 'localtime') DESC"
    ).fetchall()

it fails because the key created in post items is replaced by the key datetime(p.created, 'localtime').

I think the right way is to use a convert function like in the first post (and in the documentation). However, it would be better to reuse the local time instead of forcing to UTC.

[VMD281]

I see you've been working on this issue, and I’d like to contribute if there's any part I can help with. Have you made any progress, or is there a specific task I can assist with?

[sblondon]

I think the next step is to write a PR with a convert function providing the local timezone.
Feel free to do it if you want: I'm currently on another topic so, perhaps you can do it before I will be back on this issue.

[pawr]

By adding a default converter in the db.py, based on initial example and linked recipes, all tests pass in both venv3.11 venv3.12 using the following:

def convert_timestamp(val):
    return datetime.datetime.fromisoformat(val.decode())

def  get_db():
    if "db" not in g:
        sqlite3.register_converter("timestamp", convert_timestamp)
        ...

The convert_timestamp could apply astimezone(tz) with an optional tz=None but figured that may be out of scope for the tutorial. Otherwise, it works with fromisoformat(val.decode())

[torkian]

this small code change fix my issue :
file one change: /examples/tutorial/run_tests.py
@@ -0,0 +1,5 @@
+import unittest

+if __name__ == "__main__":
+    testsuite = unittest.TestLoader().discover('tests')
+    unittest.TextTestRunner(verbosity=1).run(testsuite)

file two  change: /examples/tutorial/tests/conftest.py
@@ -1,13 +1,22 @@
 import os
 import tempfile
-
 import pytest
+import sqlite3
+import datetime
 
 from flaskr import create_app
 from flaskr.db import get_db
 from flaskr.db import init_db
 
+# Register a custom converter for timestamps
+def convert_timestamp(val):
+    """Convert Unix epoch timestamp to datetime.datetime object."""
+    return datetime.datetime.strptime(val.decode("utf-8"), "%Y-%m-%d %H:%M:%S").replace(tzinfo=datetime.timezone.utc)
+
+sqlite3.register_converter("timestamp", convert_timestamp)
+
 # read in SQL for populating test data
+
 with open(os.path.join(os.path.dirname(__file__), "data.sql"), "rb") as f:
     _data_sql = f.read().decode("utf8")

[sblondon]

I suggest you to send a PR so the maintainers can check it and accept it directly if they agree with the patch.
I advise you to add a link to this issue so they can read the analysis.