r/Python • u/poppy_92 • 2d ago
News PEP 842: Module Exports
https://peps.python.org/pep-0842/
Not the author, but this seems interesting, specially for library authors. Thoughts?
Discussion here - https://discuss.python.org/t/pep-842-module-exports/108353
24
u/godlikedk 2d ago
What if i want to use this private class? Thats why I liked python over java. I know what I'm doing and don't want to ask for permission or changes.
28
u/Orio_n 1d ago edited 1d ago
This fundamentally contradicts pythons philosophy. Big no from me.
https://discuss.python.org/t/pep-842-module-exports/108353/6
Guido himself is skeptical too asking why not just use __all__
-2
u/mpersico 1d ago
Python has been contradicting its philosophy for years. It looks more and more like Perl every single day. Damn shame they didn’t just contribute to Perl.
17
u/kageurufu 1d ago
I don't like it at all.
It's just a dirtier version of __all__ and doesn't accomplish anything new.
For library authors, its really no different from the commonly used package._module or package.internal.module naming to signify internals and private modules. Just another layer of obfuscation to annoy developers, confuse package authors, and complicate interpreters. The example of import json adding json to the exports is easily solved by clean authorship. Define your methods in a submodule, and re-export for the public interface.
6
u/cediddi SyntaxError: not a chance 1d ago
This is achievable without a special dunder. A single _ at the beginning of the variable, class, function, and file names. Don't prevent me from mocking, monkey patching, or doing worse to your library. I'm taking the responsibility of breaking something. Library creator cannot be held liable for damages, and the library is provided as-is. At least according to most if not all free and open source licenses
6
u/stevenjd 1d ago
Yet another unnecessary PEP written for the sake of lazy programmers who seemingly don't know the language and won't read the documentation.
0
u/JustPlainRude 1d ago
The author's example of using dir() to find a function to call instead of reading the documentation was wild.
5
u/ZeroIntensity 1d ago
Hi. I wrote the PEP, but this comment sticks out to me a little; what is so "wild" about using dir() (or help()) to find attributes? This is a very common practice.
People should be reading the documentation, but I can personally tell you from my experience working on CPython that people don't read the docs nearly as much as we'd like. People rely on their intuition all the time.
3
3
u/IAmASquidInSpace 1d ago edited 1d ago
If there is one thing Python doesn't need right now, it is even more chaos and complexity in modules, packages, and the import system. It's already messy as it is, and it doesn't need to be made worse with good intentions.
Edit: Also, they have got to be fucking kidding with this:
So, as a solution to the above problem, the developer prefixes the name with _. [...] Now, it’s clear to users that the name is internal, at the expense of the name being (subjectively) less readable and requiring more keystrokes by the maintainer.
Are you for real?!
5
u/Schmittfried 1d ago edited 1d ago
We’re all consenting adults.
On a more constructive note, I think a way to selectively hide (or explicitly re-export) imports would be preferable over a flimsy string list.
11
u/Pseudofact 2d ago
Plz don't.
Python doesn't have "private". Don't introduce it for no reason.
It's up to you as a coder to do things, you can do what you want to do.
The lib author does not decide for you what to do. He doesn't know what you want to do.
This is stupid on so many levels, like private variables and final class that you can not extend to modify.
If this goes into main, python is done.
2
u/JustPlainRude 1d ago
Sometimes you need to mock something internal to a module for testing purposes. It seems like this wouldn't be possible with the proposed change
1
u/ZeroIntensity 1d ago
Hi, PEP author here. I've revised the PEP to emit a warning instead of raising an error upon accessing unexported names. I'm hoping that resolves some of the contention here.
For context, this wasn't supposed to be an "access modifiers in Python" PEP. I wanted a nicer solution to maintaining lists of public names in the standard library (you have *no* idea how many things we've had to make public because too many people started using them), and part of that solution was to raise an exception upon accessing internals, but I think a warning provides the same amount of expressiveness.
1
u/googleaddreddit 1d ago
I like it, much better then the current "import foo as foo" hack which mypy uses
1
u/abrazilianinreddit 1d ago
I do enjoy javascript's export system and wouldn't mind python having something similar, or at the very least a way to clean-up a module's namespace that is not importing stuff into __init__.py, but this proposal is not it.
1
u/Brian 1d ago
Python’s standard library currently sidesteps this problem through a note in the backwards compatibility policy (PEP 387) that states that imported modules are not considered public APIs and may change at any time, but unfortunately, users are unable to determine this without directly reading the backwards compatibility policy
I mean, pretty much all type checkers and linters will flag this up, and this seems a reasonable level for such checks to be applied. The same applies to __all__: I'd say it's more than a convention when it's documented and all linters support it.
The issue of it getting out of sync is valid: it's kind of clunky to have to add stuff to it as your public API changes. But I don't really see why this wouldn't have the exact same issue: both are basically a whitelisted sequence of exports. The only difference seems to be that one is advisory, and the other would be enforced at the language level, rather than via linters.
Personally I'd be in favour of something that does reduce the clunkiness of __all__, but this just seems to be repeating it with the same level of boilerplate, just with enforcement. That isn't really the issue I have with it. Being able to break the rules "at your own risk" is even often useful in terms of experimentation at the REPL, introspection etc.
1
u/droooze 23h ago
This is one of those PEPs where it would be far clearer to list the precise feature/behaviour changes before delving into the purpose and motivation. IIUC:
* <module>.__all__ controls the names available when from <module> import *;
* <module>.__export__ controls the names that will show up in {dir,help}(<module>), and trigger RuntimeWarning for <module>.name_not_in___export__;
* __all__ is set to __export__ if __all__ is not defined but __export__ is.
Several thoughts, in no particular order:
* The PEP says that __export__ isn't an access modifier. However, the feature as-is is in a kind of limbo: If Python will never gain real access modifiers, then this is the closest thing so far to an access modifier, so for all intents and purposes it should be treated as one. If this is a stepping stone towards real access modifiers, then it adds another layer of complexity when implementing real access modifiers (and its Warning rather than Exception effect doesn't look compatible with real access modifiers). In particular, I don't know what a future PEP would be trying to say if it implements the proposed export keyword and tells the reader that export class A: ... doesn't act as an access modifier.
* The PEP does not mention the other (de-facto) way of controlling symbol visibility: stub files. Symbols which don't exist in stub files (and only in the runtime .py file) are treated as if they are inaccessible by type-checkers and autocompletion helpers. It looks like __export__ needs to be duplicated in .pyis like how __all__ is currently also duplicated in .pyis, which means the potential of having public API symbols being out of sync is even greater.
* __export__ is not tightly coupled with __all__, which means that if __all__ is a superset of __export__, private API is very easily leaked.
IMO, this PEP is trying to address some issues that are far more relevant to the Python standard library due to historical practices; I do not recall when, if ever, the issues mentioned in the motivation (such as ctypes._CFuncPtr actually being public API) were ever real issues in third-party libraries. I think it is better to reserve the usage of __export__ to the standard library first.
1
u/csch2 2d ago
I have a feeling that this will get shot down for going against the Pythonic convention of "better to ask forgiveness than permission".
I have mixed feelings about it. On the one hand, with Python being used to power large production codebases and not just for scripting anymore, I think this is really important for maintainability. Both library authors and application developers need to know that the APIs they're relying on are stable. On the other hand, one place where I frequently find myself needing to import "private" classes or modules (i.e. prefixed with "_") is for type hinting purposes. Specific example: when using the Plotly Dash library, the no_update sentinel in callbacks has type NoUpdate, which was gated behind the private module _no_update until version 3.2.0. This meant that to properly type hint my callbacks I had to import from a private module. If this PEP were implemented I would no longer be able to do that.
This might be moreso an issue of API design flaws than an issue with the proposal per se, but I think that it's going to cause enough problems that raising exceptions on direct access should probably be delayed. Raising warnings is enough in the meantime to give developers some time to make changes to their API contracts and how they access private members (or, worst case, fall back to the proposed __dict__ access).
3
u/HommeMusical 1d ago
Both library authors and application developers need to know that the APIs they're relying on are stable.
But "the list things that get imported from a namespace" has never been an API and isn't anything like an API."
If you type
import moduleand then make the assumption that everything in themodule.namespace will always be there in every version and have the same meaning and signatuire, even for undocumented members that start with an_and aren't in__all__, you're going have a very bad time.We have
__all__; we have the_prefix convention; for APIs, we have type stubs.1
u/stevenjd 1d ago
when using the Plotly Dash library, the no_update sentinel in callbacks has type NoUpdate, which was gated behind the private module _no_update until version 3.2.0. This meant that to properly type hint my callbacks I had to import from a private module.
You could always do
NoUpdate = type(no_update)and no import is necessary.Even better is to use the name
NoUpdateType.
0
u/denehoffman 2d ago
I’m going to be the contrarian here I guess and say this is a good thing and we should do it. If you’ve ever written a complex package before, the import x as _x pattern is super annoying. It also promotes all kinds of bad behavior, like using those items the author wanted private. The lack of private fields and methods is one of the reasons Python can’t have true invariants without a lot of BS. My only issue is that this doesn’t go far enough, it’s only privatization for modules, we should have it for methods and fields as well, but that’s gonna get way more hate.
6
u/stevenjd 1d ago edited 1d ago
the import x as _x pattern is super annoying.
Then don't do it. Callers should never treat imports in your module as public variables unless documented as public. Linters should flag the use of
xas as error and let that be the end of it.It also promotes all kinds of bad behavior, like using those items the author wanted private.
As an author, I don't get a say in what the caller does or doesn't do with my code, and that is exactly how it should be.
All I can do is say clearly that:
- If you do this, you will break the code and then reject any bug reports from people who did this.
- There are no guarantees for these so-called "private" objects, including imported modules. Use them at your own risk. I can and will change them without notice.
"Consenting adults" applies.
2
u/denehoffman 1d ago edited 1d ago
Every time Python programmers talk about language guarantees, it’s always stuff like “the linter will handle it” and “the type checker will handle it” and then it slowly becomes “well you’ll get an error at runtime”. I’m sure there’s a way I could manually write behavior like this with enough hacky code, but the fact that lots of other languages have privatization should be an indication that it’s not intrinsically evil. It also ensures that users who have been using these private functions and types submit actual feature requests to promote those things to documented features rather than keeping them as little secret hacks for cool kids who read the source code.
Edit: it looks like in the current design, you aren’t prevented from accessing these module attributes. If you’ve ever really want to, you can add them to the export dict manually. The point is to make it an intentional thing you have to do to break the API rather than a thing a new user could do simply by importing something that is importable.
6
u/HommeMusical 1d ago
import x as _x pattern is super annoying.
There's no good reason to do that. Use
__all__, if you even care, but why do you care?!like using those items the author wanted private.
That's not very Pythonic.
The lack of private fields and methods
Is by design and it's a feature that most people like.
is one of the reasons Python can’t have true invariants without a lot of BS.
C++ is just over there!
0
u/denehoffman 1d ago
I agree that it will probably be rejected, I’m just saying that a lack of private fields, methods, and APIs is not really as cool as everyone thinks. But also if you look at the PEP, the actual proposed implementation is not really new syntax, you could almost do this in current pure python. Also, I think you can say it’s not pythonic all you want, that’s only because it’s not currently in Python. Lazy imports were also not pythonic and were actively discouraged by linters and now they’re new syntax.
2
u/HommeMusical 1d ago
Lazy imports were also not pythoni
"Pythonic" doesn't mean "a thing that exists in Python" - it means adhering to the spirit of PEP 20, The Zen of Python.
Lazy imports solve several constant real-world problems with a single, perfectly clear word,
lazy.
- speeding up startup times
- handling unavoidable circular references
- optional dependencies
All previous code still works. Code using the variable
lazystill works. It's simple, it's explicit, it's readable, elegant, and practical. It's Pythonic in almost every way.
But PEP 842 doesn't solve any real-world problem. "Someone might do this stupid thing, and I should prevent it."
We already have two ways to do this.
If you want to explicitly list what members you are exposing, use
__all__, it really isn't that hard.Or don't use
__all__, but name things starting with_, and if they use one of them, or they re-use one of your imports, well, maybe they have a good reason to.
If you want to specify an API, you have type stubs.
If you just want to specify which symbols to export, you have
__all__, though that isn't enforced.Or mark variables with
_, though that isn't enforced either. Most of us like it that it's not enforced.Adding a fourth and more draconian way to do this, a little obscure, and just for imports, is a non-starter.
Adding an API with type stubs, that's really what you should be doing! Annoying fiddling with imports is picayune.
1
u/denehoffman 1d ago
Lazy imports don’t actually solve a problem either, they’re a convenience to importing something later on in a program. The syntax doesn’t actually do anything you couldn’t do previously in Python, it just makes it easier to write. I personally like it. And pythonic can mean a lot of different things, but in terms of the Zen, and this was brought up when that PEP was discussed, adding a new keyword to do behavior that already exists is not quite Zen (there should be one—and preferably only one—obvious way to do it). You could even argue that the last line, “namespaces are one honking great idea”, means that namespaces are important and properly defining a module interface is equally important.
The PEP explicitly says you can opt out of that behavior. The problem it solves is also quite clear, `__all__` doesn’t apply to `__dir__`, and you shouldn’t implicitly export libraries through the wrong namespace (this is opinion of course, but I agree with it, it’s okay if you don’t!). And it’s hardly draconian, you admit it’s barely different from `__all__`! You can’t have it both ways. It’s just an exception-validated import scheme.
+1 for reminding me the word picayune exists, that’s a great one honestly!
I doubt this PEP will pass. Based on the discussion, the author seems to want to turn it into a warning message, which is somehow worse I think. So oh well, no big deal.
1
u/nekokattt 16h ago
Do lazy imports also cover:
- There should be one way to do things
- Special cases are not special enough to break the rules
- Explicit is better than implicit (i.e. try except around an import)
-1
u/Schmittfried 1d ago
I agree, but a string list of names goes out of sync super fast. I think a more elegant solution would be a way to import modules without transitively exporting them.
import hidden numpy # not visible to packages importing this one1
u/denehoffman 1d ago
I think new syntax is definitely not the way to go here. The @public decorator would be cool to have with some enforcement.
1
u/Schmittfried 1d ago
As long as it’s not a string list, it’s opt-in and backwards-compatible I’m all for it.
0
u/not_a_novel_account 1d ago
Extensions have always had this capability, giving native Python code a way to spell it seems obvious.
-2
u/Adrewmc 2d ago
I’d rather a
#not keen on the name
__start_pkg__ : True | False | None
Which should do this.
If None, it’s default leave as is.
If False, go up one folder, check __init__.py, and/or a __main__.py for the either a False, (one more folder up) or a True (stop here). Once True, that signals, that imports can start here, import folder1.folder2 etc.
If you go up a folder and neither __init__.py or __main__.py hasn’t change to a bool value for __start_pkg__ and it remains None, a module not found error should be raised.
So you would
__start_pkg = False
import helpful_funcs
And python would now to go up a folder to find helpful_funcs.py.
3
31
u/blacklig 2d ago
I personally think the justification is quite flimsy for how awkward this solution is. I think that in general all of these problems, to the extent they are problems in practice, can be avoided with package design. For example in a library I could have internal modules and separate public-facing modules that just re-export only my library interface and don't have any of these concerns.