Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement caching of NPM in CI / local dev #43167

Open
potiuk opened this issue Oct 18, 2024 · 0 comments
Open

Implement caching of NPM in CI / local dev #43167

potiuk opened this issue Oct 18, 2024 · 0 comments
Assignees
Labels
area:CI Airflow's tests and continious integration area:dev-env CI, pre-commit, pylint and other changes that do not change the behavior of the final code

Comments

@potiuk
Copy link
Member

potiuk commented Oct 18, 2024

Currently when NPM packages are installed on CI they are installed from the scratch by "compile www assets" pre-commit.

This causes fairly frequent, intermittent issues when there is a networking issue or NPM server issue - for example:

Compile www assets (manual)..............................................Failed
  - hook id: compile-www-assets
  - exit code: 1
  
  yarn install v1.22.21
  (node:490) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
  (Use `node --trace-deprecation ...` to show where the warning was created)
  [1/4] Resolving packages...
  [2/4] Fetching packages...
  [] 0/1573[] 5/1573[] 11/1573[] 18/1573[] 26/1573[] 34/1573[] 41/1573[] 45/1573[] 53/1573[] 61/1573[] 68/1573[] 75/1573[] 84/1573[] 90/1573[] 97/1573[] 104/1573[] 111/1573[] 116/1573[] 122/1573[] 128/1573[] 135/1573[] 140/1573[] 147/1573[] 153/1573[] 160/1573[] 167/1573[] 176/1573[] 185/1573[] 192/1573[] 197/1573[] 198/1573[] 202/1573[] 206/1573[] 211/1573[] 216/1573[] 223/1573[] 229/1573[] 232/1573[] 234/1573[] 238/1573[] 241/1573[] 243/1573[] 246/1573[] 248/1573[] 250/1573[] 255/1573[] 260/1573[] 265/1573[] 269/1573[] 276/1573[] 284/1573[] 286/1573[] 287/1573[] 290/1573[] 292/1573[] 295/1573[] 298/1573[] 305/1573[] 311/1573[] 316/1573[] 325/1573[] 331/1573[] 336/1573[] 342/1573[] 346/1573[] 349/1573[] 353/1573[] 359/1573[] 364/1573[] 368/1573[] 372/1573[] 378/1573[] 385/1573[] 391/1573[] 396/1573[] 402/1573[] 409/1573[] 414/1573[] 420/1573[] 427/1573[] 432/1573[] 437/1573[] 441/1573[] 443/1573[] 444/1573[] 445/1573[] 448/1573[] 451/1573[] 454/1573[] 457/1573[] 462/1573[] 467/1573[] 470/1573[] 473/1573[] 477/1573[] 481/1573[] 485/1573[] 490/1573[] 496/1573[] 503/1573[] 512/1573[] 519/1573[] 524/1573[] 526/1573[] 529/1573[] 531/1573[] 535/1573[] 539/1573[] 545/1573[] 548/1573[] 550/1573[] 553/1573[] 560/1573[] 567/1573[] 574/1573[] 580/1573[] 591/1573[] 599/1573[] 607/1573[] 616/1573[] 620/1573[] 624/1573[] 627/1573[] 630/1573[] 634/1573[] 641/1573[] 650/1573[] 658/1573[] 668/1573[] 676/1573[] 684/1573[] 692/1573error Error: https://registry.yarnpkg.com/@babel/plugin-syntax-logical-assignment-operators/-/plugin-syntax-logical-assignment-operators-7.10.4.tgz: Request failed "500 Internal Server Error"
      at ResponseError.ExtendableBuiltin (/opt/airflow/files/home/.cache/pre-commit/repocv9ljx22/node_env-22.2.0/lib/node_modules/yarn/lib/cli.js:696:66)
      at new ResponseError (/opt/airflow/files/home/.cache/pre-commit/repocv9ljx22/node_env-22.2.0/lib/node_modules/yarn/lib/cli.js:802:124)
      at Request.<anonymous> (/opt/airflow/files/home/.cache/pre-commit/repocv9ljx22/node_env-22.2.0/lib/node_modules/yarn/lib/cli.js:66218:16)
      at Request.emit (node:events:520:28)
      at module.exports.Request.onRequestResponse (/opt/airflow/files/home/.cache/pre-commit/repocv9ljx22/node_env-22.2.0/lib/node_modules/yarn/lib/cli.js:141751:10)
      at ClientRequest.emit (node:events:520:28)
      at HTTPParser.parserOnIncomingClient (node:_http_client:700:27)
      at HTTPParser.parserOnHeadersComplete (node:_http_common:119:17)
      at TLSSocket.socketOnData (node:_http_client:542:22)
      at TLSSocket.emit (node:events:520:28)
  info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
  Traceback (most recent call last):
    File "/opt/airflow/./scripts/ci/pre_commit/compile_www_assets.py", line 71, in <module>
      subprocess.check_call(["yarn", "install", "--frozen-lockfile"], cwd=os.fspath(www_directory))
    File "/usr/local/lib/python3.9/subprocess.py", line 373, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['yarn', 'install', '--frozen-lockfile']' returned non-zero exit status 1.
  
  Traceback (most recent call last):
    File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "/opt/airflow/files/home/.local/share/hatch/env/virtual/apache-airflow-build/lib/python3.9/site-packages/hatchling/__main__.py", line 6, in <module>
      sys.exit(hatchling())
    File "/opt/airflow/files/home/.local/share/hatch/env/virtual/apache-airflow-build/lib/python3.9/site-packages/hatchling/cli/__init__.py", line 26, in hatchling
      command(**kwargs)
    File "/opt/airflow/files/home/.local/share/hatch/env/virtual/apache-airflow-build/lib/python3.9/site-packages/hatchling/cli/build/__init__.py", line 82, in build_impl
      for artifact in builder.build(
    File "/opt/airflow/files/home/.local/share/hatch/env/virtual/apache-airflow-build/lib/python3.9/site-packages/hatchling/builders/plugin/interface.py", line 155, in build
      artifact = version_api[version](directory, **build_data)
    File "/opt/airflow/hatch_build.py", line 613, in build_standard
      run(cmd, cwd=work_dir.as_posix(), check=True, shell=True)
    File "/usr/local/lib/python3.9/subprocess.py", line 528, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['pre-commit run --hook-stage manual compile-www-assets --all-files']' returned non-zero exit status 1.
  Error building Airflow packages
Error preparing Airflow package

We should - similarly to Python packages, implement caching strategy that should speed up the installation of npm packages on a clean CI runner, as well as local development instance - where previously installed packages could be stored in the cache and reused. This should increase both - speed of CI jobs an stability of them (and speed of local installation of npm packages).

@dosubot dosubot bot added area:CI Airflow's tests and continious integration area:dev-env CI, pre-commit, pylint and other changes that do not change the behavior of the final code labels Oct 18, 2024
@bugraoz93 bugraoz93 self-assigned this Oct 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:CI Airflow's tests and continious integration area:dev-env CI, pre-commit, pylint and other changes that do not change the behavior of the final code
Projects
Status: Ready
Development

No branches or pull requests

2 participants