Experiences with argnames/struct

Implementers Forum - PIPS PIP-0102
1

Following the ongoing discussion on “static dicts”, aka Ciao argnames or ECLiPSe “structs”, I have implemented most of it in a branch GitHub - SWI-Prolog/swipl-devel at argnames Here are some experience that is hopefully useful input for the discussion.

Implementation

Given built-in support for dicts based on the same syntax, that was fairly trivial. As read/1 is in C in SWI-Prolog, this also means adding the basic structures for storing, importing and exporting argnames to modules, which is a bit more work.

Declaration and reflexion

(as examples we use a 3d point)

  • :- argnames(point(x,y,z)).
    Module-local declaration.
  • :- export(argnames(point(x,y,z)).
    Exported version of the same. Importing this module imports the point argnames. Introspection is possible using module_property(Module, exported_argnames(List)) as well as argnames_property(point, exported(true)) and for the importing module argnames_property(point, imported_from(the_point_module))
    discussion
    Should we (also) allow for :- module(m, [p1, argnames(point(x,y,z))]). ?
  • current_argnames(?Name, :Term).
    can be used with any instantiation.
    discussion
    Should the meta argument be on the Name or Term? The choice follows old current_predicate/2 as found in e.g. Quintus AFAIK.
  • argnames_property(:Name, ?Property)
    Can be used with any instantiation. Defined properties are arity(Int), functor(Name/Arity), exported(Bool) and imported_from(Module)

Basic use

  • point{x:1} reads as point(1,_,_), etc.
  • x of point is macro-expanded to 1, following ECliPSe. Also the ECLiPSe property(P) of point are implemented.

Experimental

  • named_arg(x, point{x:1}, X) → X = 1.
    Signature names_arg(?Name, :Term, ?Value).
    Performance comparison to arg(x of point, point{x:1}, X) is barely noticeable, unless cases where arg/3 is mapped to a VM instruction in SWI-Prolog.

Integration with SWI-Prolog dicts

  • point{x:1}.x
    Is evaluated as for dynamic dicts. All functional evaluation is supported, in part directly and in part after automatic type translation. For example:
  • ?- X = point{x:1}.put(type, '3d') → X = #{x:1, y:_, z:_, type:'3d'}
  • argnames_to_dict(point{x:1}, Dict, []) → Dict = #{x:1, y:_, z:_}.
    The option list allows for nonvar to suppress the y and z and tag(Tag) to modify the #
  • dict_to_argnames(#{x:1,type:'3d'}, point, Pt) → Pt = point(1,_,_)
    I.e., keys from the argnames that appear in the dict are used and the others are ignored.

Naming

I used argnames throughout because it is easy to replace. After this work, I think the name is acceptable for Ciao’s scope. If, however, you want to consider point(,,_) as something different from an ordinary compound this name is problematic. You can’t call the thing an “argnames” (at least, it sounds silly to me). For now I did :slight_smile: If you want to have these “things” cooperating with dicts, you have to consider them separate entities, be it “a compound with additional functionality”

Relation to dynamic dicts

This is a bit hard. “Argnames” are great for predicate heads for which you often only want to specify a few of the arguments. I start having my doubt on their usefulness as data structure. Dynamic dicts are about as fast, more flexible and easier to manage. Yes, argnames use less space, but only if more than half of the arguments are used. For _sparse _ argnames, dicts are cheaper. I think things would change if we combine them with module-local functors, such that the term can efficiently be protected.

2

Hi Jan! That is great progress.

Should we re-organize the work on dynamic and static dicts as two separate PIPs?

The discussion about naming and the relation with dynamic dicts is interesting. To me argnames does not introduce any new data structure, it is just plain old terms with a nicer syntax when naming arguments is more convenient (similar to naming arguments in other programming languages).

Some other points:

  • Ciao does not allow exporting operators from modules and we use ‘packages’ for exporting syntax, so we won’t be able to export argnames in the proposed way, although it seems reasonable. There are some nasty interactions with incremental compilation (we must think carefully about it).
  • Most of the time, we just export predicates to access the structure (but exporting the names may be useful too!)
  • Our position about the benefits of module-local functors is clear (Exploiting Term Hiding to Reduce Run-Time Checking Overhead | SpringerLink) :slight_smile:
3

I don’t know. I spent several days on this. It now basically works and should be close to what we have been discussing in the PIP meetings (except for the naming). With all that in place I start to have my doubt on the usefulness of “static dicts”/“argnames”/“structs”. A good example of using them is to access high arity predicates that act as or are a simple table. Typical examples are wrappers around database tables, database tables converted to Prolog, CSV converted to Prolog, etc. Here, writing book{author:Author, year:Year} is great to avoid counting _ while it makes changes to the data easy. But, I have done that with dynamic dicts simply by using goal_expansion/2 that translates a dynamic dict into a compound based on a declaration of the columns. You could use the same trick for writing clauses (facts) using dynamic dicts, although I think that has far fewer use cases.

The other use case is for packing data in a single data structure. Here, dynamic dicts/objects/hash-maps/… are widely used in many modern languages. Bringing that to Prolog makes Prolog more accessible and greatly simplifies connecting Prolog to deal with JSON, JavaScript, Python, etc. That was the motivation for dynamic dicts as we find them in SWI-Prolog.

I see few problems in dealing with dynamic dicts. The main issues with them are

  • The role of the “tag”. Allowing for a variable tag was possibly a bad idea. If we do not however, we should probably define that several of the predicates operating on dicts should ignore the tag, i.e.

    ?- point{x:A} :< #{x:a}.
false.

    So to extract some keys of a dict whose tag you do not care about uses _{x:X, y:Y} :< Dict, which unifies whatever tag Dict has to the anonymous variable.

  • Perhaps :</2 and >:</2 should work recursively, i.e., do partial matches on nested dicts.

For short, I have the impression we do not need static dicts/… unless we can find other good use cases for them. I’m eager to find these as it remains difficult to throw away three days of typing, some of which is a nice cleanup of old code :slight_smile: One of the options could (I think) be data hiding. I don’t think that we want a facility hiding the internals of dynamic dicts. Hiding functors is way more promising and already current practice in XSB and Ciao. Now it is still not clear to me whether it is not enough to have just dynamic dicts and a bit of goal/term expansion to map these to compounds in the scope of a module.

Yes. That is also one of my worries. I haven’t addressed this yet, but I also do no see an obvious way out.

If you do that, SWI-Prolog’s library(record), which was developed with Richard O’Keefe is fine. It simply turns :- record point(x,y). into a set of predicates that you can then export (something which you have to do individually as it is now). In practice, this doesn’t happen often. I added the export because ECLiPSe has it and I thought that was the direction the PIP meetings were heading. I stopped using library(record) after the introduction of dynamic dicts.

Can you summarise the technical part? I’ve seen the declaration is

:- hide point/3.

Now, what happens? I got the impression that this means that, in this file, point(,,_) gets a functor that does some name mangling with the module, right? From inside the module, I guess it works completely normal. What do other modules see? How does this interact with read/write, i.e., can we still serialize arbitrary data, even if the data contains hidden compounds from multiple modules?

4

Wow.

Of course, as long as you don’t have a feature, you don’t use it. I think we all agree that vanilla Prolog structures are virtually unusable in larger software projects. We probably also agree that nobody uses workarounds like library(record) because they are just no fun.
The struct-syntax introduced 25 years ago was an attempt to remedy this, and I’d say it was was pretty successful among ECLiPSe’s (admittedly small) user base. I personally would find it extremely painful to write any larger piece of code without it. So, my impression is exactly the opposite of Jan’s.

I’m surprised by these claims, they do not correspond to my experience. I have never used structs in those ways – I normally use them as I would a struct in C, or a data object.

For illustration, here are the 76 files from the ECLiPSe git (mostly libraries and compiler components) that contain 180 real life examples of structs (look for the local struct or export struct declarations):

Some good examples are compiler_common.ecl bfs.ecl time_log.ecl and then in no particular order cardinal.ecl generic_global_gac.ecl ldsb.ecl list_collection.ecl gfd.ecl compiler_regassign.ecl compiler_peephole.ecl compiler_varclass.ecl compiler_indexing.ecl compiler_codegen.ecl source_processor.ecl gnuplot.ecl mutable.ecl graphviz.ecl vc_support.ecl tty_vc.ecl vis_client.ecl lint.ecl instrument.ecl pretty_printer.ecl coverage.ecl xref.ecl r.pl set.pl mysqlopts.ecl dbi.ecl flatzinc.ecl b_trees.ecl colgen_.ecl eplex_.ecl dual_var.ecl hash.ecl best_first_search.ecl fd_domain.pl var_name.ecl tracer_tcl.pl port_profiler.ecl asm.pl constraint_pools.ecl io.pl csv.ecl branch_and_bound.pl document.ecl repair.pl ic_symbolic.ecl generic_hybrid_sets.ecl config_opts.ecl ech.pl generic_sets.ecl ic_probe_support.pl generic_sbds.ecl flow_constraints_support.ecl sd.ecl tentative.ecl ic_kernel.ecl probe_support.pl eplex_relax.pl generic_global_constraints.ecl mdd_support.ecl graph_algorithms.ecl sparse_set.ecl generic_gap_sbdd.ecl ic_constraints.ecl heap_array.ecl generic_mdd.ecl generic_gap_sbds.ecl cpviz.ecl structures.ecl vis_structures.ecl checker.ecl top.ecl checker.ecl top.ecl

(note: some of these files still use legacy syntax foo with [a:A,b:B] instead of modern foo{a:A,b:B})

5

Thanks for sharing! It is very instructive to have a look at the code of other Prolog systems. I guess something I (we?) should have done much more :frowning: I have started with C-Prolog and later worked with Quintus and a bit of SICStus, so SWI-Prolog is mostly inspired by practices developed there.

Yes, you use structs a lot! Even for really tiny data structures for which I typically use simple compounds. Only if arity gets a bit higher, the positions are not obvious (as in point(X,Y)) or it is not unlikely one may wish to extend the structure I move to dynamic dicts. I also introduced them much later (2013), so they are less abundant. Still, there are 113 files using dicts according to a simple grep (may be off a few).

Now I still think the code wouldn’t get longer or uglier when using dynamic dicts rather than structs. What the struct gives you is a set of field names and struct types. I have the unrestricted tag on dynamic dicts for the types, but the runtime does not restrict the keys depending on the type. It would be interesting to know the impact on memory usage and performance.

I’m happy to accept this as an example where statically defined structs are useful though :slight_smile:

6

| jschimpf
April 9 |

  • | - |

JanWielemaker:

I start to have my doubt on the usefulness of “static dicts”/“argnames”/“structs”.
…
I have the impression we do not need static dicts/… unless we can find other good use cases for them.

Wow.

Of course, as long as you don’t have a feature, you don’t use it. I think we all agree that vanilla Prolog structures are virtually unusable in larger software projects. We probably also agree that nobody uses workarounds like library(record) because they are just no fun.
The struct-syntax introduced 25 years ago was an attempt to remedy this, and I’d say it was was pretty successful among ECLiPSe’s (admittedly small) user base. I personally would find it extremely painful to write any larger piece of code without it. So, my impression is exactly the opposite of Jan’s.

Our impression is similar to Joachim’s. I cannot remember when ‘argnames’ were introduced in Ciao (some versions date from 28 years ago, Manuel can probably provide some archeological evidence :slight_smile: ), but argnames is definitely a language extension that is expected to appear by natural language evolution as a response to each programmer’s needs.

There are a few language extensions in Ciao that people barely use but that we use extensively in our code base and greatly simplifies the code.

JanWielemaker:

A good example of using them is to access high arity predicates that act as or are a simple table. Typical examples are wrappers around database tables, database tables converted to Prolog, CSV converted to Prolog, etc.

I’m surprised by these claims, they do not correspond to my experience. I have never used structs in those ways – I normally use them as I would a struct in C, or a data object.

For illustration, here are the 76 files from the ECLiPSe git (mostly libraries and compiler components) that contain 180 real life examples of structs (look for the local struct or export struct declarations):

Some good examples are compiler_common.ecl bfs.ecl time_log.ecl and then in no particular order cardinal.ecl generic_global_gac.ecl ldsb.ecl list_collection.ecl gfd.ecl compiler_regassign.ecl compiler_peephole.ecl compiler_varclass.ecl compiler_indexing.ecl compiler_codegen.ecl source_processor.ecl gnuplot.ecl mutable.ecl graphviz.ecl vc_support.ecl tty_vc.ecl vis_client.ecl lint.ecl instrument.ecl pretty_printer.ecl coverage.ecl xref.ecl r.pl set.pl mysqlopts.ecl dbi.ecl flatzinc.ecl b_trees.ecl colgen_.ecl eplex_.ecl dual_var.ecl hash.ecl best_first_search.ecl fd_domain.pl var_name.ecl tracer_tcl.pl port_profiler.ecl asm.pl constraint_pools.ecl io.pl csv.ecl branch_and_bound.pl document.ecl repair.pl ic_symbolic.ecl generic_hybrid_sets.ecl config_opts.ecl ech.pl generic_sets.ecl ic_probe_support.pl generic_sbds.ecl flow_constraints_support.ecl sd.ecl tentative.ecl ic_kernel.ecl probe_support.pl eplex_relax.pl generic_global_constraints.ecl mdd_support.ecl graph_algorithms.ecl sparse_set.ecl generic_gap_sbdd.ecl ic_constraints.ecl heap_array.ecl generic_mdd.ecl generic_gap_sbds.ecl cpviz.ecl structures.ecl vis_structures.ecl checker.ecl top.ecl checker.ecl top.ecl

(note: some of these files still use legacy syntax foo with [a:A,b:B] instead of modern foo{a:A,b:B})

Thanks for sharing. That is a great collection of code! I believe that we have similar use cases.