Possible to import a module as an object attribute?

[lrh9]

I'm just wondering if it is possible to import a module as an object attribute.

The basic description of what I want to accomplish is that I'm creating a software agent object. This agent will have a set of abilities (functions), but I don't know what these are ahead of time. Logically, it is simplest to encapsulate these abilities in a module, and then have the agent import it. Then it would have access to these abilities. This would also allow it to use the standard modules or any other previously developed modules. Another benefit is that each object has its own namespace for modules. If all modules were in the global namespace of the program, every agent would have access to it. This is something I do not want.

Now I obviously don't know how things work, so I resorted to perhaps one of the worse ways to code something: experimentation. The first thing I tried was to simply attempt to import a module as an object attribute. That failed. Then I checked to ensure that a module could indeed be imported to a local namespace. So I defined a class method that imported the random module, printed the local dictionary, and printed a random number. Then to ensure that the module was only accessible by that function, and not globally, I tried to print a random number in the main function. random was in the local dictionary for the test function, and it printed a random number. The main function raised an exception.

I was going to try to add the module to the local dictionary of the agent, but I learned that local dictionaries cannot be modified. I'm not even sure that an object has a local dictionary anyway. So I was thinking about giving the agent another attribute, a dictionary with holding the agent's modules. That begs two questions. First, how would one call functions using such a dictionary? Second, how do I add modules to the dictionary? I was thinking you could define a method that sets the agent's module attribute as the result of a function that imports a given module, and returns the local dictionary. However, I'm inclined to believe this won't work. I believe objects in a function's local namespace cease to exist when that function exits.

I tested my last assumption, and apparently you can return the locals of an object's method and store them in an attribute. I forgot that locals() returns a copy of the local dictionary. Now I just need to resolve how to call functions from the module name in the dictionary. I'll go ahead and post this while I'm working on it. Maybe someone will come up with an answer before or I do, or maybe I'm going about this the wrong way or there is a better way to do it.

Thank you for reading this. Any help will be appreciated. Thank you.

[lrh9]

To access functions in a module in the module dictionary, you simply use:

In an object's method:

self.modules[module_name].function(arguments)

Externally:

x = Agent()
x.modules[module_name].function(arguments)

Where "module_name" is a string or variable containing a string that is the module's name, or the name the module was assigned on import. "function" is the name of the function in the module, and "arguments" are the arguments to the function. I just posted this to let everyone know it can be done. I'll refine my code and post it so everyone can have a clearer and more complete picture of how it works.

[Gribouillis]

Is this what you mean ?

class Agent:
    def __init__(self):
        exec "import random" in self.__dict__


ag = Agent()

print ag.random.randrange(100) # this works

print random.randrange(100) # this doesn't work (random not in global namespace)

You could even make a class decorator

def withrandom(cls):
    exec "import random" in cls.__dict__
    return cls

@withrandom
class Agent:
    pass


ag = Agent()

or simpler

class Agent:
    import random

[pythopian]

Sounds like your agents are plugins with no pre-defined interface. To load such modules, you can use this snippet.

Obviously, you can't just go modules[module_name].function() in your case, because function is not known in advance. I guess you'll have to inspect the module. Pythons inspect module comes in handy:

import inspect

# using function importPluginModulesIn from snippet at
# http://www.daniweb.com/code/snippet238387.html

modules = importPluginModulesIn('myagents')
for name, module in modules.items():
    functions = inspect.getmembers(module, inspect.isfunction)
    classes = inspect.getmembers(module, inspect.isclass)
    print( '\nModule %s has %d classes and %d functions' % (name, len(classes), len(functions)) )
    print( 'Classes:', classes )
    print( 'Functions:', functions )

Output would be something like:

Module foo has 1 classes and 1 functions
('Classes:', [('Foo', <class foo.Foo at 0x021D4DE0>)])
('Functions:', [('dofoo', <function dofoo at 0x021D2BF0>)])

Module bar has 1 classes and 1 functions
('Classes:', [('Bar', <class bar.Bar at 0x021D4DB0>)])
('Functions:', [('dobar', <function dobar at 0x0217C4F0>)])

Similarly, you can capture the module's contents into an object dict:

# using function importPluginModulesIn from snippet at
# http://www.daniweb.com/code/snippet238387.html

class Agent:
    def __init__(self, module):
        vars(self).update(vars(module))

agents = [ Agent(module) for module in importPluginModulesIn('myagents').values() ]
# now the module's contents are accessible via an agent
agents[0].dofoo()
agents[1].dobar()

[lrh9]

Looks good. It looks better than what I wrote. I'll have to give it all a look over though.

Here's what I did.

#!/usr/bin/env python

from uuid import uuid4 as _uuid4
from importlib import import_module as _import

class Agent:

    def __init__(self, name = None):
        self._uuid = _uuid4()   # Universally unique identifier.
        if name is None:
            name = self.uuid
        self.name = name
        self._modules = {}      # Agent's modules dictionary.


    def _get_uuid(self):
        return self._uuid

    uuid = property(_get_uuid)  # help(property)


    def _get_name(self):
        return self._name

    def _set_name(self, name):
        self._name = name

    def _del_name(self):
        self._name = self.uuid

    name = property(_get_name, _set_name, _del_name)


    def add_module(self, name, package = None):
        self._modules[name] = _import(name, package)

    def _get_modules(self):
        return self._modules

    def _set_modules(self, modules):
        self._modules = modules

    def _del_modules(self):
        self._modules = {}

    modules = property(_get_modules, _set_modules, _del_modules)

def main():
    try:
        bond = Agent()  # Software agents, not secret agents.
        print(bond.name)
        bond.name = "James Bond"
        print(bond.name)
        print(bond.uuid)
        print(bond.modules)
        bond.add_module('random')
        print(bond.modules['random'].random())
        trevelyan = Agent("Alec Trevelyan")
        print(trevelyan.name)
        print(trevelyan.uuid)
        trevelyan.modules = bond.modules
        print(trevelyan.modules)
        print(trevelyan.modules['random'].random())
        del trevelyan.modules
        print(trevelyan.modules)
    finally:
        pass

if __name__ == "__main__":
    main()

[lrh9]

Sorry about all that code. I was in a rush. Next time I'll post only the relevant sections of the source.

[lrh9]

Is this what you mean ?

class Agent:
    def __init__(self):
        exec "import random" in self.__dict__


ag = Agent()

print ag.random.randrange(100) # this works

print random.randrange(100) # this doesn't work (random not in global namespace)

Oh. "__dict__" is the object's dictionary. Thank you for telling me that.

That was the same idea I had at first, but I couldn't find it. However, I think my solution is slightly more flexible because I can create a new agent with the modules of a previous agent. In your solution, I couldn't copy the object's dictionary because each agent is supposed to have at least one unique value. Thank you for trying, and you did help.

[lrh9]

Sounds like your agents are plugins with no pre-defined interface. To load such modules, you can use this snippet.

I just wanted to see if it could be done. I haven't thought of how someone would interface with it yet.

[lrh9]

I've been doing some more research and testing, and I've hit a roadblock. While an object can have a reference to an imported module, it doesn't actually have its own copy of the module. Somehow Python only imports the module once, and somehow each and every agent object points directly to this module. That means modifying the variables in one agent's module modifies those variables in agent's with the same module, and reloading one agent's module reloads it for all agent's.

I know of no easy way to alter this behavior. I'm thinking I'm going to have to try to write my own importer to achieve the results I desire.

[Gribouillis]

I've been doing some more research and testing, and I've hit a roadblock. While an object can have a reference to an imported module, it doesn't actually have its own copy of the module. Somehow Python only imports the module once, and somehow each and every agent object points directly to this module. That means modifying the variables in one agent's module modifies those variables in agent's with the same module, and reloading one agent's module reloads it for all agent's.

I know of no easy way to alter this behavior. I'm thinking I'm going to have to try to write my own importer to achieve the results I desire.

The standard pattern in python is that when you want different copies of similar things, you don't use modules but class instances. I think your agents should aggregate objects having the desired abilities instead of modules with a non standard importing policy. Another possible design is to dynamically add methods to your agents (some kind of mixin stuff).

Anyway, this is too theoretical. you should write some clear use cases for your 'agents'. There is most probably a better design for this.

[lrh9]

The class wrapper for a module is a brilliant idea. It might even be dynamically generated so that it would support all modules compatible with the Python version for it.

What are mixins? I get the general abstract that a class composed of mixins is an object that pools similar and related functionality, but you might need to clarify and expound.

I have a general use case, or else I wouldn't have a plan for a program, but how do I arrive at a clear and specific use case? The system I'm working on is general purpose and dynamic. I've only recently begun working with Python. Don't get me wrong. I've become technically proficient and I know my way around code and documentation - so I don't need to learn the mechanics of Python - but I don't always no the correct way to handle the computer science or software design. For instance, my idea to create a custom import system - all though technically correct - was incorrect in terms of computer science and software design.

[lrh9]

I did some more research, and after learning that a module is in fact an object and learning about introspection, I think I finally came up with a decent solution.

Setting a classes dictionary to that of another classes dictionary effectively makes it a copy of that class. Correct? So creating a dummy class and setting its dictionary to that of another module's dictionary effectively makes it a copy of that module. Correct? Using the dict() function to copy the dictionary is a true copy, instead of creating a reference. So here is the final solution I wrote.

class Module_Replicator:

def __init__(self, module_object):
    self.__dict__ = dict(module_object.__dict__)

import random

mymodcopy = Module_Replicator(random)

print(random.BPF)
print(mymodcopy.BPF)
mymodcopy.BPF = 1
print(random.BPF)
print(mymodcopy.BPF)

Now I'm sure it's not a perfect solution. So what if anything should I do differently? For one thing, my object and the module object isn't the same thing. Meaning that comparing the two object's for sameness will return false. How do I fix this? One plausible solution I considered - but have not tested - is in the initialization of the object setting the self object as the module object and then performing a true copy of the module's dictionary. Does anyone see any other problems?

[lrh9]

I did some quick testing, and my idea of setting the self object to a module won't work. It makes the class's __dict__ attribute a readonly attribute. Which I find strange, because I remember from earlier experimentation storing a new variable in a module object.

[pythopian]

I've been doing some more research and testing, and I've hit a roadblock. While an object can have a reference to an imported module, it doesn't actually have its own copy of the module. Somehow Python only imports the module once, and somehow each and every agent object points directly to this module. That means modifying the variables in one agent's module modifies those variables in agent's with the same module, and reloading one agent's module reloads it for all agent's.

I know of no easy way to alter this behavior. I'm thinking I'm going to have to try to write my own importer to achieve the results I desire.

I did some quick testing, and my idea of setting the self object to a module won't work. It makes the class's __dict__ attribute a readonly attribute. Which I find strange, because I remember from earlier experimentation storing a new variable in a module object.

Have you overseen my solution? (See above, 3rd code block.) It solves your problem and circumvents the "read-only-ness" of __dict__.

[lrh9]

Have you overseen my solution? (See above, 3rd code block.) It solves your problem and circumvents the "read-only-ness" of __dict__.

I looked it over, but apparently load_source is obsolete. I realize it will still be in the imp module, but there is no telling when they will deprecate and remove it.

After looking over the imp module, I don't think it will be too difficult to implement a custom import solution that fulfills my specification.

The imp module has a function called new_module which returns a new module.

According to PEP 302 at the end of the section "Specification part 1: The Importer Protocol", the way a module object is created is a empty module object is created and then the module file's code is executed in the module object's dictionary.

This simple code will load the random module from a Python 3.1 installation on the C:\ drive.

import imp

file_stream = open(r'c:\python31\lib\random.py')
module_code = file_stream.read()
random = imp.new_module('random')
exec(module_code, random.__dict__)
print(random.random())

This code is very basic, so there are some differences between this code and Python's implementation of import. However, more code can be added to complete additional functions of Python's import.

[pythopian]

I looked it over, but apparently load_source is obsolete. I realize it will still be in the imp module, but there is no telling when they will deprecate and remove it.

Yes, imp.load_source has been said to be "obsolete" since at least python 1.5, but it's still alive and well. Moreover, the function has never been made to issue a deprecation warning, which is the usual way to sort out python functionality. It may be that is goes away some day, but I think that day is in a far distant future, and there will be plenty of time to adapt when (and if) it comes.

[lrh9]

Yes, imp.load_source has been said to be "obsolete" since at least python 1.5, but it's still alive and well. Moreover, the function has never been made to issue a deprecation warning, which is the usual way to sort out python functionality. It may be that is goes away some day, but I think that day is in a far distant future, and there will be plenty of time to adapt when (and if) it comes.

I'm all ready fairly on my way in implementing a custom importer that removes the import once features of the standard Python import.

Here's the thread I started in case you missed it.

[pythopian]

Here's the thread I started in case you missed it.

Where is "here"???

[lrh9]

http://www.daniweb.com/forums/thread243780.html