Documentaton / Typing for python module as stubs / pyi files to be picked up by IDE for autocompletion and type checking
Most of the ovito module is implemented as C extensions; scripting in ovito would be more approachable if the documentation were also included as stub files, so IDE's like Spyder/VSCode could pick it up and display it, upon inspection or for autocompletion, and language servers like pylance or mypy could type check it.
For instance, when doing 'import ovito; ovito.scene.[TAB or whatever triggers autocomplete]' nothing is displayed.
Yes, I agree. We should work on making the Python API of OVITO fully discoverable and accessible to introspection. In fact, this has been on my to-do list already for a couple of months now, but I was busy with too many other dev tasks.
Let me look into possible solutions asap. Most of the classes are exported by the C++ code via pybind11. In addition, a thin wrapper layer amends some classes with additional methods and properties written in Python. I'll need to check whether stub/pyi files are really needed in this case (I am not familiar with them). Originally, my expectation was that pybind11 can generate all necessary information on the fly.
I'll keep you posted and come back to you here as soon as I have worked out a plan. Let me know if you have further advice or suggestions. Thanks.
-Alex
As promised, I now looked into possibilities to support code auto-completion for the Ovito module in Python IDEs such as VS Code (pylance language plugin). As far as I understand it now, these IDEs do not support runtime introspection of Python modules, which means they cannot directly access type information and function signatures of the C extension library. Instead, they either can do static Python code analysis (not possible in case of C extension modules) or they require stub files (.pyi).
I looked into ways to generate .pyi stub files for pybind11 extension modules, but non of the solutions that are currently available (e.g. mypy stubgen, pybind11-stubgen) yields acceptable results. It seems the only option left is writing handcrafted stub files for all classes and functions in the ovito package. I started doing that after first reorganising the internal structure of the ovito package and the way it imports the pybind11 class bindings from the C extension module.
You can have a look at first results of my work by installing the following development version of the PyPI package in your Python interpreter: https://pypi.org/project/ovito/3.7.2.dev1058/#files
It contains stub files for all ovito.* sub-modules except for the ovito.modifiers module. The latter is still on my to-do list. Furthermore, I wrote tooling to automatically extract the Sphinx docstrings from the C extension module and merge them with the handcrafted stub files. This is done in order to not only give you autocompletion in the IDE but also online documentation of classes and functions.
It would be great if you could give me some feedback. If this is going in the right direction, I should be able to integrate this into the next official OVITO release.
Thanks for the fast turn-around!
I just tested the linked dev version, autocomplete works on VSCode and Spyder.
My next test is converting some manual processes to a script ( editing the result of 'convert to script' ), using the stub files IDE-provided documentation.
I'll let you know how that goes, but so far seems to work.
Here's an example of what a typed python modifier looks like:
from typing import Optional, TypeVar
from ovito.data import CutoffNeighborFinder
from ovito.modifiers import ComputePropertyModifier
import numpy as np
from ovito.data import DataCollection
T=TypeVar('T')
def notnone(x:Optional[T])->T:
assert x is not None
return x
def modify(frame : int, data : DataCollection):
p = np.array(notnone(data.particles).positions)
print(p.shape)
p = p.reshape((-1,6,3)).transpose((1,0,2))
va = notnone(data.cell).delta_vector(p[0],p[2])
vb = notnone(data.cell).delta_vector(p[5],p[3])
va/=np.linalg.norm(va,axis=-1)[...,None]
vb/=np.linalg.norm(vb,axis=-1)[...,None]
d = np.sum(va*vb,axis=-1)
# Output the computed per-particle entropy values to the data pipeline.
notnone(data.particles_).create_property('Dot', data=np.repeat(d,6))
Attached screenshot from vscode. Not much is shown, but pylance is running and correctly autocompletes / typechecks the code, once the fact most objects may return None is handled.
Uploaded files:
Thanks for the feedback.
Yes, the thing with optional return objects is something I noticed too. If a property that is typed as returning Optional[SomeObject] is accessed, VS Code doesn't show the available member fields of SomeObject in the autocompletion list - sometimes at least. I couldn't really figure out when this happens and why. Sometimes pylance appears to have concluded that the return value is not None, and then the autocompletion works as expected. I see in your example that you are explicitly forcing this to happen with your notnone() function.
So while the initial stub code I wrote for the DataCollection class looked like this:
class DataCollection(DataObject):
@property
def particles(self) -> Optional[Particles]: ...
I later changed it to:
class DataCollection(DataObject):
@property
def particles(self) -> Particles: ...
to make the autocompletion behaviour in VS Code more convenient for the user. You find this new stub definition in the latest official release 3.7.3 of the ovito package.
In practice, DataCollection.particles is almost never None - except when creating an empty DataCollection; then it is None indeed. The same reasoning applies to many other fields such as Particles.positions or DataCollection.cell.
What is your opinion on this? Should we put the focus more on convenient autocompletion behaviour or more on correctness of the interface definition? In any case, I'm not sure if my handcrafted stub files are comprehensive enough to facilitate static correctness checking.
Since Datacollection.particles/etc is None only on initialization I think anyone interested can quickly write a factory function that will limit the 'incorrect' interface to a small section of code. Other static type checks are useful, like just yesterday the restriction on the global attributes being floats/strs spared me some debugging.
Just updated to 3.7.3. This being on the official release is great, I can start showing it to my colleagues! I also tested with jupyter lab's documentation tab, works too. Thanks for the quick work!
A quick note for anyone exploring the documentation on vscode this way, e.g. doing 'cell.__class__' will show the class documentation.
新的OVITO微信频道!
New for our users in China: OVITO on WeChat
Official OVITO WeChat channel operated by Foshan Diesi Technology Co., Ltd.
