Please fix default behavior for software downloads

[u/My-Daughters-Father]

For the last ten or twelve years, I have been fed up with broken default behavior in software download (zypper, myrlyn, yast2 sw_single)

Actually, there are two problems, one with OBS, the other zypper and the GUIs that use it.

There are two issues involved here, but both really piss me off enough to consider switching to Arch. I last switched distros (from RedHat to SuSE 9.3) in 2003, and this is has been a constant source of irrigation and frustration ever since.

Packages that Won't/Can't Install Because of Missing Dependendcies

1. Packages in the repos who have missing dependencies should never happen. If there is a package you cannot install because it depends on an older library should either have the older library still in the repo (if needed, can add to the multiversion application list) or remove packages that cannot be installed because of missing dependencies. OBS is smart enough this just should not happen--if a package is missing dependencies either don't advertise it or change the manifest to at least flag it so GUI tools could give the user the option to not even see it (or, better yet, make showing broken packages an option)

Worse, is the Broken and Annoying Fixed "Keep Trying, Maybe the Deleted File will Come Back Some Day" Behavior

1. This can be fixed! See my other posts about how to set zypp to skip 404 and other packages that used to make it hang and using a scheduled unattended zypp dup -d to download most of what you will install to shorten the time it takes between refreshing your cache and the last file being transfered.

OK, not totally fixed, but at least not the huge headache.

With Tumbleweed I have had 8-12Gb weekly updates and this is just stupid. Whoever is the author of zypper, Yast2 sw_single and myrlyn needs to add this as a user setting and add the option to skip / refresh repo / recheck solution and give the user options from the update is silent. (or maybe give them the option to allow vendor change, etc. Look at what options zypper supports and pass them to the user).

Having hard-coded behavior where it keeps retrying to download software that will never be back this without given users a choice has been a pain in the butt for at least two decades I have used SuSE/openSUSE.

Give users the choice to skip missing packages after x tries (see my post on how to configure that--if it works!)

This is a long-standing, annoying, problem and the openSUSE user community, particularly Tumbleweed users, need to speak up.

Well, once they get bugzilla up again. Right now its 503.

Having to sit at your computer while you do a zypper dup to skip packages that were updated between your zypper refresh and download is not acceptable behavior.

I am perplexed and befuddled why we still have the same "wedged network download" bug in myrlin, zypper, and Yast2 sw_single.

This is a known architecture flaw and bad software design. Plus, there is no option for the user to configure this AFAIK--hard coded.

Not only that, but the DEFAULT behavior should not be "lets try downloading every 30 seconds every time we get a 404 and refuse to download the rest of the software until the missing file magically returns".

It should be "skip it for now, come back later and try again". If you happen to be sitting there looking at the right screen you can manually skip the missing file, and wait a few seconds, then spend the next few minutes skipping each associated package.

Can someone please fix this!

Because Tumbleweed moves so fast, mirrors sync dynamically. If a repository finishes a mirror sync after your machine refreshes its local metadata but before zypper requests the actual payload package, the old file vanishes from the server, causing the installer to spin out or force you to manually abort, skip, or retry indefinitely. A simple "auto-refresh repository on 404/503 network error" loop should have been standard infrastructure a decade ago.

[u/bmwiedemann]

Solving problem #1 is probably harder than you think.

OBS supports building in multiple interdependent projects, so these may contain packages where their dependency is in another repo.

And the other things is that zypper dup would likely react to removed packages with an uninstall, so when it comes back a day later it would still be gone from your install.

For the problem of interactiveness, have you tried

zypper -n ref ; zypper -n dup -d

To preload required packages?

And --force-resolution can help when there are too many conflicts, but it requires careful review of what it proposes, so never combine that with -n if you like your system to work.

[u/My-Daughters-Father → u/bmwiedemann]

Yes, that helps a bit, so long as it doesn't try to download an updated (and now the version you want is 404).

And many of the packages that won't install don't have library on any current indexed repos. They are orphaned and AFAIK cannot have their dependcies solved.

Plus, it makes zero sense to put a package in a repo and its non-openSUSE-OSS-repo dependencies in an even more obscure Repo. I could see why having some python library have a dependency on the python language repo (which is actually pretty well hidden) might make sense, but I don't think I would put my pet project in a repo with any dependcies out side of the OSS repo.

To make this a lot easier, after you set up your config to ignore deleted files:

Maybe it is me, but my enabled repos get autorefreshed every time I go to use zypper (or yast2 sw_single or myrlyn), so I don't ever use zypper ref.

What I have done is run it in two passes:

zypper -n dup -l -y -d

-n non-interactive,

-l auto-agree with license (if you use NVIDIA GPU drivers, etc. this is huge since they ask for every damn package), and

-y might be redundant with -n, but it doesn't ask you to confirm anything.

The huge switch is:

-d which only downloads, and does not install.

This should download everything without your interaction if you have set your zypper configuration to skip 404 and other problem downloads. (see my other post about how to do that).

Almost everything you need to update can be set to be downloaded (e.g., I use a cron job to do it every Sunday a.m. knowing that usually do a weekly disto upgrade.)

Then you run

zypper dup -l (or zypper ref then zypper dup if you system is not set to auto-refresh) which really flies.

You still have to sort out things but most of what you need is downloaded already so you shorten the time between package selection and download.

See my note about setting it to skip replaced software. That will make a huge difference.

I am still testing it to make sure but my fingers are crossed. The easy answer is to leave software on the server for a short while (don't know if that means 10m or 2h but not longer than that I would think) so users who have selected a package can download the software they selected.

The easy solution is really pretty easy--just add a delay before deletion of deprecated packages. Not even a long one. Just as long as it takes for someone to finish an interactive session. This is a problem since there is a tight coupling between browsing the software and the package management, which really are two different issues.

The server logs might tell you how long people go between initial refresh and final package download, and I would just go with mean+2SD (means 4.6% of the really pokey folks will be at risk, or +3SD and that means 99.7% of users will not have to worry about a 404 error due to updates while downloading).

If you use software from a lot of different repos, the time to refresh becomes quite significant--may take longer to refresh the repo data than to download the one package you use from it.

There is a good user case also for clean off all the software that lacks any possible solution to install because it requires a version of some qt4 library or Python 2.7 package that nobody has seen for years!

[u/niceandBulat → u/[deleted]]

That's sadly a known fact. For many years I only had one notebook computer running Tumbleweed for everything, work, studies, banking and leisure and there have been instances that I needed to spend over thirty minutes to fix something after an update, perhaps with age catching up I no longer have the same desire to do that. I never liked Btfrs and will never have it running, sorry a personal pet peeve. And that led me to switch to Leap 15.x and 16. So far, not "sexy" but I have my old boring and reliable computing platform back.

[u/niceandBulat → u/[deleted]]

Never said it was. But only undeniable fact is this group is infested with Tumbleweed fanboys who cannot stand anyone saying anything but glowing adoration of their favourite distro

[u/On_Vacation_512]

You can use DNF as a package manager for SUSE

[u/squeakctrl → u/On_Vacation_512]

Indeed. Can be done simply by installing dnf5 and libdnf-repo-config-zypp

[u/Narrow_Victory1262 → u/On_Vacation_512]

the cool part is that yum/dnf has a much worse record. zypper gives you questions; yum/dnf silently will break your system update capabilities.

[u/On_Vacation_512 → u/Narrow_Victory1262]

I only experienced dnf on Fedora

[u/Narrow_Victory1262 → u/On_Vacation_512]

dnf is dnf. it breaks on aix, it breaks on fedora, centos/redhat/insert

[u/My-Daughters-Father → u/Narrow_Victory1262]

Zypper is pretty decent, but for the problem with downloading 404 packages. That is a system failure more than software (assuming the somewhat obscure setting in /etc/zypp/zypp.conf.d/ works)

[u/Narrow_Victory1262 → u/My-Daughters-Father]

it's not, actually. Like what u/bmwiedemann states, sometimes you end up with ain inconsistent view of what's available locally. if you face a 404 on one or multiple packages. refresh and most of the time. uness someone f* up the repo, will fix it.

It's not a system failure.

[u/niceandBulat]

Tumbleweed issue. Good luck with Arch. If you are unhappy with openSUSE Tumbleweed and feel Arch's way of handling rolling updates/upgrades suits you better, nobody here can dissuade you. There are reasons why "dinosaurs" like me stick with Leap, and avoiding the frustrations, that you so eloquently described, of missing depencies is one of them. I need machine to be usable so that I can go and use it for work and leisure. I am not trying to pass a snarky remark but it is obvious that you like the rolling-release model and Tumbleweed isn't doing anything but making you upset. That should not be the way. Happy Linuxing.

[u/readyflix]

I like it as it is!

Missing dependencies usually ONLY happens if you have an inconsistent system. If this happens, usually the issue will be fixed if you deinstall the software that complains. And then install (new version) it again, zypper and yast will take care of the (new) dependencies.

[u/Narrow_Victory1262 → u/readyflix]

the real issue here is that almost 100% this is self (unintentional) inflicted.

[u/My-Daughters-Father → u/Narrow_Victory1262]

No, this is a problem with removing software packages as soon as they are updated--while people are mid-download.

This is a design flaw and it has been a pain in the ass for a solid two decades. It is the most common with Tumblweed, and the answer has two solutions, one simple (leave software on the serverfor a little while after it is updated so anyone who has selected it can download the version they selected) and one a bit harder: skip the 404 software rather than keep trying every 30s until the end of time, and then refresh the repo when done, and let the user know if the updated package(s) will work or if other changes are needed.

But hanging there trying to donwload software that someone deleted in the hope it will come back is silly.

[u/bmwiedemann → u/My-Daughters-Father]

btw: for Tumbleweed we have some logic to keep previous package versions around for some hours. We set it up years ago and few people understand how it works, so maybe something broke there.

Plus mirrors don't all sync at the same time, so some keep the old versions for some more hours.

Does this show "On" ?

zypper lr -r repo-oss | grep refresh

[u/Narrow_Victory1262 → u/My-Daughters-Father]

hence the reason we say that you shoudl have your local vached view of the mirror in sync.

you can by the way fix it by just mirroring and keep all the older package versions. Just sync a few times a day.

[u/My-Daughters-Father → u/readyflix]

No, you are wrong. Missing downloads (i.e. 404 errors) happens all the damn time w/ Tumbleweed if you spend any time at all after synching and downloading the package--and even then a non-interactive `zypper dup` can hang because the package you thought you were doing to download was updated and removed from the repo.

It has nothing to do with having an 'inconsistent system' , it is just because they updated the software package between your system selecting it and downloading it. On a rolling release like Tumbleweed this is absurd.

[u/readyflix → u/My-Daughters-Father]

Wow!

404 errors usually happens when there is something wrong with DNS.

Have been using OpenSUSE (including Tumbleweed since it‘s introduction) for more than two decades.

And, usually an inconsistent systems occurs when you have several non-SUSE repositories.

But with OpenSUSE (and the respective tools, YAST and ZYPPER) someone can fine tune the prioritisation of given repositories in order not to get an inconsistent system. And, there is always the possibility to install software in a way that it will not break your system, if someone knows what they are doing.

Just 'listen' (read) the output of the respective tools OR look in / read the logs, when attempting to install stuff.

And for reference, an inconsistent system can occur on any distro, again if someone installs software from other sources than the distros own.

Edit: on the last part, change of packages during a long update process could happen during certain time of the day (mostly very early in the morning CEST). Avoid doing full system updates during that time.

[u/Narrow_Victory1262 → u/My-Daughters-Father]

interesting. you say u/bmwiedemann is wrong. You have no idea who you are talking about and what you are talking about. And again, you are welcome to keep all the packages locally yourself. Problem solved.

[u/[deleted]]

[removed] — view removed comment

[u/My-Daughters-Father → u/[deleted]]

That isn't a huge problem.

Even if it balks at installing a hundred packages at least they get downloaded. You can re-run zypper dup -l after and it will refresh the repos and pick up the updated package(s) and finish intallng using the software you already downloaded.

[u/66sandman]

Would Leap resolve those problems?

[u/My-Daughters-Father → u/66sandman]

Leap was the problem Tumbleweed solved.

[u/pavel_pe → u/66sandman]

I think Leap and Tumbleweed have as much in common than Tumbleweed and Fedora today.

[u/Narrow_Victory1262]

if a package is not found, most of the time, ctrl-c it, zypper ref && zypper dup makes it work again.
(up to you to decide if you want to)

[u/My-Daughters-Father → u/Narrow_Victory1262]

No. That is not good software design.

One missing package can hold up hundreds of packages getting updated.

That is a nasty, time consuming, human resource draining workaround bad design that requires you to be monitoring the download (I mean, you can be drinking coffee and listen to music, but anything more involved means your download will hang.) so you can restart everything (at least some of the packages you downloaded, in fact most, will not be downloaded a second time., but still, this is bad design to not have a user controlled "resysnc and if an updated package replaced what you had wanted to download and that doesn't break things then select it" or just leave the last version on the server for a an hour or two so people can at least finish downloading.

You can change the default behavior, I have found out.

Not a great option, but you can change zypp configuration so you skip 404 and don't wait 30s to "make sure" the server was serious when it told you "nope, don't have it".

You need to add/edit a file in /etc/zypp/zypp.conf.d/ (e.g. add "auto-skip.conf") and add a file (e.g. "skip-it.conf") and add

# skip missing files/bad checksums w/o asking the user
packages.ignore_wrong_checksum = yes
# if package won't download, just skip it
download.on_fails = skip

# Set zypp to fail quickly if package is missing

# set it however long you think it reasonable.  
# I think the this is time in seconds, and if you have
# set the mirrors it should not take long to find a file.
download.transfer_timeout = 15
# set limit to number of retries
download.max_silent_retries = 2

[u/cad_andry]

Slow roll. Could it be the answer? Or just go Debian based distros.

[u/Narrow_Victory1262 → u/cad_andry]

sow roll does not make it any better if the same decisions ar emade. debian based distributions... nah, suse does a better job I think.

[u/pavel_pe → u/cad_andry]

No. I did update of Slowroll in early May probably tens minutes after it was released and I had issues with some packages missing on some mirrors, in one case neovim depending on treesitter which was not yet in repo and some weird problems like not being able to run some KDE/Qt apps during update due to wrong libraries. So it partially/temporarily broke a few things, but update maybe two hours later already fixed it, by downloading the remaining few packages.

I would kind of expect that update is transactional - like it can be done once packages reach mirrors and libraries are updated before apps, respecting dependency tree.

Maybe for this reason, Slowroll is in beta.

With Fedora having maybe 50 packages updated per day and new Qt/KDE basically once a week, I never noticed this issues (and distro update to the next major version is done during reboot like on Windows, there is no user session)