Make stack traces more comprehensive yet simple

[omkar-foss]

Description

As per users' feedback in the Airflow Debugging Survey 2024, user feedback around incomplete stack traces is as follows:

• Stack traces do not always show the full path of the source file, which can lead to confusion with files that have the same name across installed packages.
• Stack traces for failing DAGs often lack complete information when displayed in the UI's red error bar.
• Stack traces are generally helpful but can be misleading when they arise from dependency conflicts after upgrades.

Use case/motivation

Goals for this issue are the following:

• Automatically dump relevant variables and environment states at the time of failure, including task parameters and Airflow configurations. Ensure to cleanup the dump to remove redundant content.
• Add time information to stack traces, detailing the time spent on different operations within a task.
• Include other DAG execution state information that could be useful to the user. Preferably keep it configurable to user can disable if not required.
• Attempt to always show the full path of the source file, in order to avoid confusion during debugging.

Related issues

Parent Issue: #40975

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct