Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

brew info should display download size #18373

Open
1 task done
osalbahr opened this issue Sep 22, 2024 · 24 comments
Open
1 task done

brew info should display download size #18373

osalbahr opened this issue Sep 22, 2024 · 24 comments
Labels
features New features help wanted We want help addressing this

Comments

@osalbahr
Copy link
Contributor

Verification

Provide a detailed description of the proposed feature

I want to know the total download/install size when installing a package.

What is the motivation for the feature?

I am using a limited storage system (16 GB). Also boredom and I want to contribute to Homebrew again.

How will the feature be relevant to at least 90% of Homebrew users?

Sort of? It would be nice. But probably not needed. Storage is cheap. It will only affect people with pay-per-use data plans.

What alternatives to the feature have been considered?

brew deps then somehow find tarball size for each package. This is good enough for my use.

Example (only first step):

$ echo $(brew deps fastfetch)
binutils gcc gmp isl libmpc lz4 mpfr xz zlib zstd

Then a bash for-each, since formula names are guaranteed to not have spaces (I think).

But would be nice if it's integrated into Homebrew under an undocumented flag. I would appreciate pointers on how to start this, such as existing open issues/discussions. Thx!

More info:

$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: No developer tools installed.
Install Clang or run `brew install gcc`.
$ brew config
HOMEBREW_VERSION: 4.3.23-53-g29c22e0
ORIGIN: https://github.com/Homebrew/brew
HEAD: 29c22e0ab3fd9826cefd4b14c143266dea3bc391
Last commit: 2 days ago
Core tap JSON: 22 Sep 11:36 UTC
HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: :0
HOMEBREW_EDITOR: /usr/bin/nano
HOMEBREW_MAKE_JOBS: 8
HOMEBREW_SORBET_RUNTIME: set
Homebrew Ruby: 3.3.4 => /var/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/3.3.4_1/bin/ruby
CPU: octa-core 64-bit sandybridge
Clang: N/A
Git: 2.46.1 => /bin/git
Curl: 8.6.0 => /bin/curl
Kernel: Linux 6.10.10-200.fc40.x86_64 x86_64 GNU/Linux
OS: Fedora release 40 (Forty)
Host glibc: 2.39
/usr/bin/gcc: N/A
/usr/bin/ruby: N/A
glibc: N/A
gcc@11: N/A
gcc: N/A
xorg: N/A
@osalbahr osalbahr added the features New features label Sep 22, 2024
@osalbahr
Copy link
Contributor Author

Here is a minimal way to find the total tarball sizes:

for P in $(brew deps fastfetch)
do
    brew cat "$P" | \
    grep -m1 url | \
    awk '{print $2}' | \
    xargs wget -qO- | \
    wc -c
done

Note: tapping and downloading is required by the above for loop which (sort of) defeats the points, but this is merely a proof of concept. I think there should be a "size" field right below the sha256 value in a formula. Also theoretically a "security" measure to check both the shasum and size, but that's a later decision.

@carlocab
Copy link
Member

carlocab commented Sep 22, 2024

I think there should be a "size" field right below the sha256 value in a formula.

That's a bit tricky since the bottle contains a copy of the formula, so changing the formula changes the size of the bottle too.

That said, I think the bottle size (both compresses and uncompressed) is available in the annotations in the manifest, so in principle this should be straightforward to recover with curl and jq. I don't think it contains source tarball size, but most users probably don't really care about this (because they install formulae from bottles).

Or, in a pinch, if you only care about compressed size:

brew fetch foo
wc -c "$(brew --cache foo)" # bottle size

brew fetch --build-from-source foo
wc -c "$(brew --cache --build-from-source foo)" # source tarball size

This makes it so that you don't need to download foo all over again if you want to install it or build it from source.

@osalbahr
Copy link
Contributor Author

That said, I think the bottle size (both compresses and uncompressed) is available in the annotations in the manifest, so in principle this should be straightforward to recover with curl and jq. I don't think it contains source tarball size, but most users probably don't really care about this (because they install formulae from bottles).

I agree. The size of bottle is what I was looking for rather than the tarballs.

That said, I think the bottle size (both compresses and uncompressed) is available in the annotations in the manifest

That sounds like what I was looking for. Is there perhaps some sort of API that lets me "fetch" only the size, without first downloading the bottle?

And wc -c "$(brew --cache foo)" does seem to print the intended "Download Size" of a specific formula. But to my understanding I have to have the bottle cached locally first.

@carlocab
Copy link
Member

That sounds like what I was looking for. Is there perhaps some sort of API that lets me "fetch" only the size, without first downloading the bottle?

Yes. The manifest can be fetched separately from the bottle. (In fact, brew downloads the manifest first before downloading the bottle.)

And wc -c "$(brew --cache foo)" does seem to print the intended "Download Size" of a specific formula. But to my understanding I have to have the bottle cached locally first.

Yes, that's why you need to brew fetch first.

@osalbahr
Copy link
Contributor Author

Where can I read more about getting only the manifest? I couldn't find it documented in https://docs.brew.sh.

@carlocab
Copy link
Member

It's not documented, but you can see what brew does to fetch it by doing brew fetch --verbose:

❯ brew fetch --force --verbose libuv
==> Downloading https://ghcr.io/v2/homebrew/core/libuv/manifests/1.48.0
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.3.23-54-gde9848d\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 15.0\)\ curl/8.7.1 --header Accept-Language:\ en --fail --retry 3 --header Accept:\ application/vnd.oci.image.index.v1+json --header Authorization:\ Bearer\ QQ== --remote-time --output /Users/carlocab/Library/Caches/Homebrew/downloads/8ee1d27fb604f55e3c4415b96e34dc9c7f557996832c372d984c8162b29a4ad2--libuv-1.48.0.bottle_manifest.json.incomplete --location https://ghcr.io/v2/homebrew/core/libuv/manifests/1.48.0

@osalbahr
Copy link
Contributor Author

Now I'm thinking I perhaps should first work on a brew fetch --manifest-only foo. What do you think?

@carlocab
Copy link
Member

Seems to have a relatively niche/obscure use-case at the moment, so it's probably not a good fit.

@osalbahr
Copy link
Contributor Author

Ok. I'll see what I can do. Thx!

@umnikos
Copy link

umnikos commented Sep 22, 2024

One time I tried to install a calculator with brew. It took a weirdly long amount of time, and then I saw this:

Installing libqalculate dependency: xorg-server

I think I'd personally benefit from brew calculating some kind of size for the operation (download size should be easy) and then giving the user a yes/no prompt if that size is large.

@osalbahr
Copy link
Contributor Author

osalbahr commented Sep 22, 2024

I am no longer interested in solely working on a PR. It turned out to be more complicated than I expected as there is no API to fetch the download/install size.

But happy to help. @umnikos if you (or someone else) are interested, lmk, and I will re-open the issue. Happy to help you debug too, just @ me.

I will also consider working on this given enough thumbs up.

Best proxy for interest I could find is a 2-year-old StackOverflow post with 7 upvotes: How to make Brew show the size of the formula before installing it?.

That post also has workarounds that seem to no longer work but might be close enough. Most notably, it talks about the Bearer Token which is needed for /usr/bin/env /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Linuxbrew/4.3.23-56-g9160445\ \(Linux\;\ x86_64\ Ubuntu\ 22.04.5\ LTS\)\ curl/7.81.0 --header Accept-Language:\ en --fail --retry 3 --header Accept:\ application/vnd.oci.image.index.v1+json --header Authorization:\ Bearer\ QQ== --remote-time --output /home/linuxbrew/.cache/Homebrew/downloads/0f02a3a463ce4e72f92871751d9ba7b872ca8090348074d46ffb523fd67e1c7b--xz-5.6.2.bottle_manifest.json.incomplete --location http (the token is the main reason why simply curl doesn't work, but I do not understand the rest of that one-liner).

@osalbahr osalbahr closed this as not planned Won't fix, can't repro, duplicate, stale Sep 22, 2024
@MikeMcQuaid
Copy link
Member

brew calculating some kind of size for the operation (download size should be easy)

@umnikos if you read the whole thread you'll see why this is not easy. If you disagree: we'd welcome said easy PR 😉

t turned out to be more complicated than I expected as there is no API to fetch the download/install size.

@osalbahr as mentioned by @carlocab above: said API is "download the manifest first and use that to figure out the size of the formula".

The longer-term solution here would be to consider if we could e.g. download all these manifests daily and provide a JSON file of the rough sizes that could be incorporated into the formulae.brew.sh JSON API. I'd imagine ~24h outdated information here would be more useful than none at all.

@osalbahr osalbahr reopened this Sep 23, 2024
@MikeMcQuaid MikeMcQuaid added the help wanted We want help addressing this label Sep 23, 2024
@MikeMcQuaid MikeMcQuaid changed the title brew install: Download/Install Size brew install should display download size Sep 23, 2024
@osalbahr
Copy link
Contributor Author

osalbahr commented Sep 23, 2024

@osalbahr as mentioned by @carlocab above: said API is "download the manifest first and use that to figure out the size of the formula".

I tried to work on that, but the path for going from libuv (for example) to the long curl one-liner seemed intimidating. My approach is usually to hack something up in a language I know, like bash, then figure out how to put that in a language I am not familiar with, like Ruby. However I think I might need to go at this issue in Ruby directly.

Could you point me at the Ruby function that creates the curl command that downloads the manifest? It is difficult to figure that out since I couldn't find info on the manifest in the technical docs.

@apainintheneck
Copy link
Contributor

apainintheneck commented Sep 24, 2024

It's not exactly what you're looking for but @cho-m opened a proof of concept PR (#18172) a couple weeks ago to add formula size information in brew info. The size information gets pulled from the manifest which seems similar to what you all are talking about here. Of course, it doesn't incorporate that information into brew install but maybe there's some logic that could be shared from that PR if/when it gets merged in. At the very least it seems to be tangentially related to this discussion.

@cho-m
Copy link
Member

cho-m commented Sep 24, 2024

I did have another related P.O.C. (local idea, not a PR) of creating a summed size. Mainly was experimenting with it as part of --dry-run (and also hacking that into some user prompting feature), e.g. rough mock up for install size (download size would just be summing f.bottle.bottle_size)

--- a/Library/Homebrew/install.rb
+++ b/Library/Homebrew/install.rb
@@ -284,17 +285,27 @@ module Homebrew
           end
         end

-        if dry_run
+        if dry_run || prompt
           if (formulae_name_to_install = formulae_to_install.map(&:name))
             ohai "Would install #{Utils.pluralize("formula", formulae_name_to_install.count,
                                                   plural: "e", include_count: true)}:"
             puts formulae_name_to_install.join(" ")

+            formulae = Set.new
             formula_installers.each do |fi|
               print_dry_run_dependencies(fi.formula, fi.compute_dependencies, &:name)
+              formulae.add(fi.formula)
+              formulae.merge(fi.compute_dependencies.flatten.map(&:to_formula))
             end
+            installed_sizes = formulae.map { |f| f.bottle.installed_size }
+            ohai "Total size: #{disk_usage_readable(installed_sizes.sum)}" if installed_sizes.all?(&:positive?)
+          end
+          return if dry_run

Anyway, will try to get around to my original PR first. Hopefully will have some more time.

EDIT: Also, should now have more data to work with as all Sequoia bottles will have size information in manifest. Older versions won't.

@osalbahr
Copy link
Contributor Author

osalbahr commented Sep 24, 2024

@cho-m awesome! Let me know if you'd like some help, feel free to @ me in the current or a new PR. We're not doing the exact same, but we can build off of a common ground. I like @MikeMcQuaid's idea of adding bottle sizes in JSON API. What is your current hack, and do you have a long-term plan?

@apainintheneck thx for linking the related PR!

@osalbahr
Copy link
Contributor Author

osalbahr commented Sep 24, 2024

Oh, disk_usage_readable looks like a cool function. But I first need to figure out how to get the size_in_bytes.

@cho-m btw, how did you get installed_sizes.sum? I can't find it in the main fork or your fork (or global GitHub search). Perhaps you haven't pushed it yet or it's in a private fork.

Edit: ok, just realized it's a variable in the diff you pasted. I should probably read a Ruby book or do an online course (or ask ChatGPT). I welcome recommendations. All I'm getting so far is that the absolute values are some way of Ruby to do a lambda-style for-each and then the sizes are somehow added up. Looks like you're almost there!

@MikeMcQuaid
Copy link
Member

@cho-m @osalbahr Note: if/when we're adding more information to the API here: it may be nice to also include sh.brew.path_exec_files from the tab/manifest, too; that'll make https://github.com/homebrew/homebrew-command-not-found much nicer.

@osalbahr osalbahr changed the title brew install should display download size brew info should display download size Oct 13, 2024
@osalbahr
Copy link
Contributor Author

osalbahr commented Oct 13, 2024

Since I am just learning Ruby, I think brew info is an easier milestone.

@cho-m what do you think?


This is something I wanted to know today, just out of curiosity, on WSL (Ubuntu) brew install ruby is how much more/less storage compared to apt/snap install ruby.

$ brew info ruby
==> ruby: stable 3.3.5 (bottled), HEAD
Powerful, clean, object-oriented scripting language
https://www.ruby-lang.org/
Installed
/home/linuxbrew/.linuxbrew/Cellar/ruby/3.3.5 (19,856 files, 59MB) *
  Poured from bottle using the formulae.brew.sh API on 2024-10-13 at 12:55:08
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/r/ruby.rb
License: Ruby
==> Dependencies
Build: autoconf ✘, pkg-config ✘, rust ✘
Required: libyaml ✔, openssl@3 ✔, gperf ✔, libffi ✔, libxcrypt ✔, zlib ✔
==> Options
--HEAD
        Install HEAD version
==> Caveats
By default, binaries installed by gem will be placed into:
  /home/linuxbrew/.linuxbrew/lib/ruby/gems/3.3.0/bin

You may want to add this to your PATH.

Emacs Lisp files have been installed to:
  /home/linuxbrew/.linuxbrew/share/emacs/site-lisp/ruby
==> Analytics
install: 59,960 (30 days), 186,061 (90 days), 711,848 (365 days)
install-on-request: 22,271 (30 days), 74,546 (90 days), 313,322 (365 days)
build-error: 13 (30 days)

@MikeMcQuaid
Copy link
Member

Since I am just learning Ruby, I think brew info is an easier milestone.

@osalbahr Feel free to start with just that.

@osalbahr
Copy link
Contributor Author

Since I am just learning Ruby, I think brew info is an easier milestone.

@osalbahr Feel free to start with just that.

I appreciate the green light. It lets me know that I am moving in the right direction. Though I don't mind if someone beats me to it.

I created a repo learn Ruby. I will come back to adding the brew info feature later. Or, I might script a working solution in Bash first. I will see.

@osalbahr
Copy link
Contributor Author

I am trying to add a line under ==> Analytics that says bottle-size: 0 as a placeholder. However I can't figure out where the lines under Analytics come from. Could you help by pointing at which Ruby file is responsible for the list?

I looked at info.rb and I am not sure where the delegation happens.

$ brew info fastfetch | tail
==> Options
--HEAD
        Install HEAD version
==> Caveats
Bash completion has been installed to:
  /home/linuxbrew/.linuxbrew/etc/bash_completion.d
==> Analytics
install: 13,275 (30 days), 31,864 (90 days), 65,133 (365 days)
install-on-request: 13,275 (30 days), 31,864 (90 days), 65,133 (365 days)
build-error: 1 (30 days)

@ZhongRuoyu
Copy link
Member

That happens here:

output_analytics(json, args:)

Invoked from here in info.rb:

Utils::Analytics.formula_output(formula, args:)

@stephbowie
Copy link

/start

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
features New features help wanted We want help addressing this
Projects
None yet
Development

No branches or pull requests

8 participants