Make Airflow stack traces configurable and easy-to-understand

[omkar-foss]

Description

As per users' feedback in the Airflow Debugging Survey 2024, 55.2% respondents find stack traces challenging.

Use case/motivation

Goal for this issue:

• Configure depth of stack trace using a config property like AIRFLOW_STACK_TRACK_DEPTH which can then only show stack trace for that much depth e.g. if AIRFLOW_STACK_TRACK_DEPTH=3 then we show only 3 levels deep stack trace as the stack unwinds.
• Either depth as above or can use number of lines as well (this will be more of an absolute config as in some case 1 level deep trace can take up all the lines!).
• Use color coding wherever necessary, and simpler language for the user to understand the stack trace - probably by eliminating any redundant parts of the stack trace wherever possible.

Related issues

Parent Issue: #40975

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct