[Import-SIG] pkgutil.extend_path

[Eric V. Smith]

I'm looking at fixing pkgutil.extend_path in order to support namespace
packages where some portions use PEP-420 and some use extend_path.

The first thing I notice is that there are no tests for
pkgutil.extend_path :(

extend_path currently just examines the filesystem directly, which means
it doesn't support portions in zip files or other finder/loaders.

But if I understand PJE and others correctly, the idea is to modify
extend_path so it calls the path_hook finders instead of looking at the
filesystem (in order to find the other __path__ entries). This looks
like a change in functionality: previously only real filesystem packages
would be found. With this change it would include other finders (like
zip files). While it's a change, it would align well with importlib.
Personally I'm okay with this change.

There's the added issue of how to deal with .pkg files. Is only
supporting them from the filesystem okay? Or is it worth the hassle of
creating some finder API to access them?

Any thoughts?

Eric.

[PJ Eby]

Post by Eric V. Smith
I'm looking at fixing pkgutil.extend_path in order to support namespace
packages where some portions use PEP-420 and some use extend_path.
The first thing I notice is that there are no tests for
pkgutil.extend_path :(
extend_path currently just examines the filesystem directly, which means
it doesn't support portions in zip files or other finder/loaders.
But if I understand PJE and others correctly, the idea is to modify
extend_path so it calls the path_hook finders instead of looking at the
filesystem (in order to find the other __path__ entries).

Well, I actually wasn't trying to support that at all; I've never used
extend_path() myself, so I was only talking about implementing transitional
support for the PEP in declare_namespace().

That being said, it sounds like adding support in extend_path() might also
be worthwhile. Presumably, it would *not* auto-update, since extend_path()
doesn't currently do that. (Just as declare_namespace() *should*
auto-update, because it currently does.)

Post by Eric V. Smith
This looks
like a change in functionality: previously only real filesystem packages
would be found. With this change it would include other finders (like
zip files). While it's a change, it would align well with importlib.
Personally I'm okay with this change.

Makes sense to me.

There's the added issue of how to deal with .pkg files. Is only

Post by Eric V. Smith
supporting them from the filesystem okay? Or is it worth the hassle of
creating some finder API to access them?

I don't know who actually uses them, but then I don't know who uses
extend_path(), period. The only examples I was able to find with Google
and Nullege are of the form "try: declare_namespace() except:
extend_path()" -- that is, code that first tries to use declare_namespace()
instead.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20120515/0d3b0c2d/attachment.html>

[Eric V. Smith]

On Tue, May 15, 2012 at 1:26 PM, Eric V. Smith <eric at trueblade.com
I'm looking at fixing pkgutil.extend_path in order to support namespace
packages where some portions use PEP-420 and some use extend_path.
The first thing I notice is that there are no tests for
pkgutil.extend_path :(
extend_path currently just examines the filesystem directly, which means
it doesn't support portions in zip files or other finder/loaders.
But if I understand PJE and others correctly, the idea is to modify
extend_path so it calls the path_hook finders instead of looking at the
filesystem (in order to find the other __path__ entries).
Well, I actually wasn't trying to support that at all; I've never used
extend_path() myself, so I was only talking about implementing
transitional support for the PEP in declare_namespace().

I'm mostly interested in extend_path() because:
a) it's in the standard library
b) if it can be made to work, I assume declare_namespace() can, too

That being said, it sounds like adding support in extend_path() might
also be worthwhile. Presumably, it would *not* auto-update, since
extend_path() doesn't currently do that. (Just as declare_namespace()
*should* auto-update, because it currently does.)

Right. I have extend_path() working with mixed namespace packages: those
that have an __init__.py with extend_path, and those with no
__init__.py. In this case, it does not support auto-updating (although
it easily could, if we expose that from _bootstrap). I'll update the PEP
to mention returning the portions iterable even if a loader is returned,
clean up the code, and check it in.

I'll leave declare_namespace() for someone else. I might have time to
look at it if/when the PEP is accepted, but no promises.

There's the added issue of how to deal with .pkg files. Is only
supporting them from the filesystem okay? Or is it worth the hassle of
creating some finder API to access them?
I don't know who actually uses them, but then I don't know who uses
extend_path(), period. The only examples I was able to find with Google
extend_path()" -- that is, code that first tries to use
declare_namespace() instead.

I haven't touched the .pkg code. It doesn't work terribly well if a
zipimporter is used, but then it never did. At least I'm not making it
worse.

Once this is checked in, I think the work on the PEP and sample
implementation is done.

Eric.