Better readability and searchability of Airflow logs

[omkar-foss]

Description

As per users' feedback in the Airflow Debugging Survey 2024:

• Enhanced readability and search features are needed
• More than 50% respondents find logs difficult to search and filter

Use case/motivation

Goals for this issue are the following:

• Review existing log messages for verbosity and readability.
• Explore addition of a fuzzy search functionality in Airflow UI so user preferably doesn't have to leave Airflow window to search for Airflow logs (in other places like ES). This will enhance dev productivity.
• Explore keeping log verbosity configurable at DAG-level. For e.g. A user may want more verbosity for a model training DAG while very less verbosity for a scheduled DAG performing basic data transformations.

Related issues

Parent Issue: #40975

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct