path: root/guile-zig-local/README.md

# Guile & Zig

Zig is a blazingly fast new programing language that adhers to a stable ABI and that makes it suitable to link against other programming languages. Zig comes withouth a garbage collector, and that makes it even better.

This repo contains code to show how we can link Zig with Guile - but, really, zig directly supports any language that allows for a C FFI.

## Install

To create an environment with GNU Guix you can run with [guix.scm](guix.scm).

```
guix shell -C -D -f guix.scm
zig version
guile -v
```

This way guile can be run in an emacs shell with M-x shell -> guile.

## Compile zig

Run zit source tests with

```
zig test my.zig
```

Create libmy.so and show exported symbols with:

```
zig build-lib -dynamic my.zig
nm -D --demangle libmy.so
```

## Print zig hello world from C

The quick way is to create a C function that calls a zig function (no guile yet):

@1
```
zig build-lib -dynamic my.zig
zig cc `pkg-config --cflags guile-3.0` test.c -fPIC libmy.so $GUIX_ENVIRONMENT/lib/libguile-3.0.so -o test
env LD_LIBRARY_PATH=.:$GUIX_ENVIRONMENT/lib ./test
hello world
```

The `LD_LIBRARY_PATH` tells the runtime to look in the current working directory for mylib.so.

## Print zig hello from guile

Now we have a shared lib it should be easy to load.

A shared library can be loaded into a running Guile process with the function load-extension. See the [guile documentation](https://www.gnu.org/software/guile/manual/html_node/Combining-with-C.html). In addition to the name of the library to load, this function also expects the name of a function from that library that will be called to initialize it:


// gcc `pkg-config --cflags guile-3.0` \
//  -shared -o libmy.so -fPIC .c

```
zig cc `pkg-config --cflags guile-3.0` test.c -fPIC libmy.so -o test
env LD_LIBRARY_PATH=. guile
```

```guile
(load-extension "libmy" "init_module")
```
Should loads without error. Next we can call a function after we add a mapping to the init_module. What we want to do is:

```guile
(hello_zig)
possibly unbound variable hello_zig
```

Hello stranger.

We need to do a bit more work to make guile accept zig calls. An quick example in C is [here](https://www.gnu.org/software/guile/manual/html_node/A-Sample-Guile-Extension.html#A-Sample-Guile-Extension). libguile.h contains a range of C macros to facilite bindings. C macros (shudder - we want to get rid of those). GNU Guile is a dynamic language - that means values carry information about their type at runtime. This requires mapping values back and forth. This is true for any dynamic language - including Python, Ruby, R, etc.

One fun aspect of GNU Guile is that almost all its functions are implemented in C with a scm_ prefix. So, if you look up functions, you'll the the guile call and the C call next to each other. In the Guile manual you'll find more information on [translating types](https://www.gnu.org/software/guile/manual/html_node/Dynamic-Types.html). You'll find a C example:

```C
SCM my_incrementing_function (SCM a, SCM flag)
{
  SCM result;

  if (scm_is_true (flag))
    result = scm_sum (a, scm_from_int (1));
  else
    result = a;

  return result;
}
```

that shows that the SCM macro represents different types of parameters in Guile. SCM is defined in scm.h saying `SCM values can hold proper scheme objects only.'

If we add this function to test.c we can see its expanded version as

```
zig cc -E -I$GUIX_ENVIRONMENT/include/guile/3.0 test.c libmy.so -L$GUIX_ENVIRONMENT -lguile -o test
```

note the `-E` switch. The function expands to

```C
SCM my_incrementing_function (SCM a, SCM flag)
{
  SCM result;

  if ((!((((((scm_t_bits) (0? (*(volatile SCM *)0=((flag))): (flag))) & ~(((scm_t_bits) (0? (*(volatile SCM
 *)0=(((SCM) ((((((1)) << 8) + scm_tc8_flag)))))): ((SCM) ((((((1)) << 8) + scm_tc8_flag)))))) ^ ((scm_t_bi
ts) (0? (*(volatile SCM *)0=(((SCM) ((((((0)) << 8) + scm_tc8_flag)))))): ((SCM) ((((((0)) << 8) + scm_tc8_
flag)))))))) == (((scm_t_bits) (0? (*(volatile SCM *)0=(((SCM) ((((((1)) << 8) + scm_tc8_flag)))))): ((SCM)
 ((((((1)) << 8) + scm_tc8_flag)))))) & ((scm_t_bits) (0? (*(volatile SCM *)0=(((SCM) ((((((0)) << 8) + scm
_tc8_flag)))))): ((SCM) ((((((0)) << 8) + scm_tc8_flag))))))))))))
    result = scm_sum (a, scm_from_int32 (1));
  else
    result = a;

  return result;
}
```

There is a bit of work going on, but mostly it is simple packing and unpacking described [here](https://www.gnu.org/software/guile/manual/html_node/Relationship-Between-SCM-and-scm_005ft_005fbits.html). To use this in Zig we'll need to translate the C macros for some conversions. Note that `scm_sum` and `scm_from_int` are simple function calls. The real challenge is converting SCM to boolean with `scm_is_true`.

Zig comes with a utility that does that:

@2
```
zig translate-c -I$GUIX_ENVIRONMENT/include/guile/3.0/ $GUIX_ENVIRONMENT/include/guile/3.0/libguile.h > test-include.zig

```

And, interestingly translates all of that including

```zig
pub const scm_t_timespec = struct_timespec;
pub const scm_t_off = i64;
pub const scm_t_signed_bits = isize;
pub const scm_t_bits = usize;
pub const struct_scm_unused_struct = extern struct {
    scm_unused_field: u8,
};
pub const SCM = [*c]struct_scm_unused_struct;
pub const scm_tc8_flag: c_int = 4;
pub const scm_tc8_char: c_int = 12;
pub const scm_tc8_unused_0: c_int = 20;
pub const scm_tc8_unused_1: c_int = 28;
pub const enum_scm_tc8_tags = c_uint;
pub const scm_t_subr = ?*anyopaque;
pub const struct_scm_dynamic_state = opaque {};
pub const scm_t_dynamic_state = struct_scm_dynamic_state;
pub const struct_scm_print_state = extern struct {
    handle: SCM,
    revealed: c_int,
    writingp: c_ulong,
    fancyp: c_ulong,
    level: c_ulong,
    length: c_ulong,
    hot_ref: SCM,
    list_offset: c_ulong,
    top: c_ulong,
    ceiling: c_ulong,
    ref_vect: SCM,
    highlight_objects: SCM,
};
pub const scm_print_state = struct_scm_print_state;
pub const struct_scm_dynstack = extern struct {
    base: [*c]scm_t_bits,
    top: [*c]scm_t_bits,
    limit: [*c]scm_t_bits,
};
```

that may look familiar to Guile hackers and matches the C

```C
typedef struct scm_unused_struct { char scm_unused_field; } *SCM;
```

typed struct * that is SCM. Now from Zig, unless we start using real Guile internals we may get away with simply using our own opaque pointer that is passed around.

Let's try compiling this conversion and it gives one error

```sh
zig test test-include.zig
test-include.zig:6522:30: error: use of undeclared identifier 'sched_priority'
pub const __sched_priority = sched_priority;
```

Commenting out that line and the compile passes. That must be too good to be true(!)
Let's try compiling test.c with

```
zig cc `pkg-config --cflags guile-3.0` test.c -fPIC libmy.so $GUIX_ENVIRONMENT/lib/libguile-3.0.so -o test
env LD_LIBRARY_PATH=$GUIX_ENVIRONMENT/lib:. ./test
Hello world from 3 to 4
```

Now we add the zig call without `scm_is_false`

```
zig build-lib -dynamic my.z
env LD_LIBRARY_PATH=$GUIX_ENVIRONMENT/lib:. ./test
Hello world from 3 to 4
 from 3 to 4
```

This worked! With reintroduced `scm_is_false` we get

```
./test-include.zig:6119:62: error: unable to evaluate constant expression
pub inline fn scm_is_true(x: anytype) @TypeOf(!(scm_is_false(x) != 0)) {
```

Remember that is a C macro converted automatically. If we use a C function it works:

```
zig build-lib -dynamic my.zig
zig cc `pkg-config --cflags guile-3.0` test.c -fPIC libmy.so $GUIX_ENVIRONMENT/lib/libguile-3.0.so -o test
env LD_LIBRARY_PATH=$GUIX_ENVIRONMENT/lib:. ./test
Hello world from 3 to 4
Hello world from 4 to 5
```

Now let's try loading guile again

```
env LD_LIBRARY_PATH=. guile
```

and

```
(load-extension "libmy" "init_module")
(hello_zig)
possibly unbound variable hello_zig
```

while

```
nm -D libmy.so
000000000000aa80 T hello_zig
000000000000ab40 T my_incrementing_zig_function
000000000003d6c0 W __zig_probe_stack
```

shows the symbols are exported. Hmm. We need to tell guile first there exists a function. Apparently I am not the [first](https://github.com/ziglang/zig/issues/1675). After a bit of try and error with zig 0.9 we get:

```zig
pub export fn ping_zig(i: SCM) SCM {
    return i;
}

export fn init_module() void {
    // var c_ping_zig = @intToPtr(*anyopaque, @ptrToInt(&ping_zig)); <- ampersand for zig 0.10 and after
    var c_ping_zig = @intToPtr(*anyopaque, @ptrToInt(ping_zig));
    var res = guile.scm_c_define_gsubr("ping_zig", 1, 0, 0, c_ping_zig);
    _ = res;
}
```

which runs in guile with

```guile
(load-extension "libmy" "init_module")
(ping_zig 0)
$1 = 0
(ping_zig "TEST")
$2 = "TEST"
```

Woot! That works great. Note we did not have to deal with a type system.

With a recent version of zig the function pointer needs an ampersand.

## Lists

The work horse of data structures in Lisp is the list. Let us try to find an element that equals 2 and increment the value from Zig. First attempt

```zig
export fn my_increment_in_list_zig(lst: SCM) SCM {
    var lst2 = guile.scm_memv(guile.scm_from_int(2), lst);
    return lst2;
}
```

```guile
scheme@(guile-user)> (load-extension "libmy" "init_module")
scheme@(guile-user)> (incr_list_zig '(3 4 2 "test"))
$1 = (2 "test")
```

Now let's increment with

```zig
export fn my_increment_in_list_zig(lst: SCM) SCM {
    var lst2 = guile.scm_memv(guile.scm_from_int(2), lst);
    var lst3 = guile.scm_list_set_x(lst2, 1, guile.scm_from_int(3));
    return lst3;
}
```

Now this segfaults. Let's try the debugger with

```
env LD_LIBRARY_PATH=. gdb --args guile
```

and we get

```
Thread 1 "guile" received signal SIGSEGV, Segmentation fault.
0x00007ffff7f02dc5 in scm_to_uint64 () from /gnu/store/qlmpcy5zi84m6dikq3fnx5dz38qpczlc-guile-3.0.8/lib/libguile-3.0.so.1
```

ah, I think the index 1 should be a SCM value. Indeed does the job

```zig
export fn my_increment_in_list_zig(lst: SCM) SCM {
    var lst2 = guile.scm_memv(guile.scm_from_int(2), lst);
    lst2 = guile.scm_list_set_x(lst2, guile.scm_from_int(0), guile.scm_from_int(3));
    return lst; // note original is updated!
}
```

```guile
(load-extension "libmy" "init_module")
(incr_list_zig (list 3 4 2 "test"))
$1 = (3 4 3 "test")
```

## Arrays

Arrays are continuous in memory. Guile provides a special page on [accessing arrays from C](https://www.gnu.org/software/guile/manual/html_node/Accessing-Arrays-from-C.html).

It says: Accessing Arrays from C.
For interworking with external C code, Guile provides an API to allow C code to access the elements of a Scheme array.
In particular, for uniform numeric arrays, the API exposes the underlying uniform data as a C array of numbers of the relevant type.
This can be very useful. Note that the C and Zig code need to protect and array when it starts modifying it. And for multithreaded access locking is also required. See above documentation for more.

We'll want to create an [unboxed vector](https://www.gnu.org/software/guile/docs/docs-2.2/guile-ref/Uniform-Numeric-Vectors.html#Uniform-Numeric-Vectors) of double. This is the scheme version:

```guile
(use-modules (srfi srfi-4))
(define v #f64(3.1415 2.71 1.0))
(f64vector-length v)
$28 = 3
(f64vector-set! v 1 3.71)
v
$29 = #f64(3.1415 3.71 1.0)
```

The zig version:

```zig
// Increment the second value in the unboxed vector
export fn my_increment_in_f64vector_zig(vec: SCM) SCM {
    _ = guile.scm_f64vector_set_x(vec, guile.scm_from_int(1), guile.scm_from_double(3.7));
    return vec; // note original is updated!
}
```

does it

```guile
(define v #f64(3.1415 2.71 1.0))
(incr_f64vector_zig v)
$1 = #f64(3.1415 3.7 1.0)
```

Now in the final step we want to update the value as a raw Zig array. The
C Function:

```C
const double * scm_f64vector_elements(SCM vec, scm_t_array_handle *handle, size_t *lenp, ssize_t *incp)
```

should return a pointer. The Zig translated version is

```zig
pub extern fn scm_f64vector_elements(uvec: SCM, h: [*c]scm_t_array_handle, lenp: [*c]usize, incp: [*c]isize) [*c]const f64;
```

This code unboxes the double vector:

```zig
export fn my_increment_in_f64vector_zig(vec: SCM) SCM {
    // _ = guile.scm_f64vector_set_x(vec, guile.scm_from_int(1), guile.scm_from_double(3.7));
    var handle: guile.scm_t_array_handle = undefined;
    var lenp: usize = undefined;
    var incp: isize = undefined;
    var data = guile.scm_f64vector_elements(vec, &handle, &lenp, &incp);
    p("\nZig says {any},{any},{any}\n",.{data[0],data[1],data[2]});
    return vec; // note original is updated!
}
```

When you try to mutate it will complain! We need to set the vector to mutable. Replacing
the line with:

```
var data = guile.scm_f64vector_writable_elements(vec, &handle, &lenp, &incp);
```

gets a complaint:

```
Wrong type (expecting mutable f64vector): #f64(3.1415 2.71 1.0)
```

Try again by createing a mutable vector. This is the full code

```zig
export fn my_increment_in_f64vector_zig(vec: SCM) SCM {
    // _ = guile.scm_f64vector_set_x(vec, guile.scm_from_int(1), guile.scm_from_double(3.7));
    var handle: guile.scm_t_array_handle = undefined;
    var lenp: usize = undefined;
    var incp: isize = undefined;
    var data = guile.scm_f64vector_writable_elements(vec, &handle, &lenp, &incp);
    p("\nZig says {any},{any},{any}\n",.{data[0],data[1],data[2]});
    data[1] += 1.0;
    guile.scm_array_handle_release(&handle);
    return vec; // note original is updated!
}
```

```guile
(load-extension "libmy" "init_module")
(define v (list->f64vector '(3.1415 2.71 1.0)))
(display "Guile says ")
(display v)
(incr_f64vector_zig v)
(display "Guile now says ")
(display v)
```

displays

```
Guile says #f64(3.1415 2.71 1.0)
Zig says 3.1415e+00,2.71e+00,1.0e+00
Guile now says #f64(3.1415 3.71 1.0)
```

## Sharing a pointer

If you want to move pointers around, both guile and zig can handle 'foreign pointers'. Zig has *anyopaque and guile has [foreign pointers]( https://www.gnu.org/software/guile/manual/html_node/Foreign-Pointers.html#Foreign-Pointers).