1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
|
# Binding notebooks
Binder and Cocalc are tools for deploying Jupyter notebooks, R/shiny notebooks and Pluto notebooks. These binding tools do a lot more and do not leverage GNU Guix lightweight container infrastructure. In this project we want to create a binding-lite.
## Tags
* keywords: binder, jupyter, notebooks
* status: unclear
* priority: medium
* assigned: jgart, pjotrp
## Members
* jgart
* pjotr
## Kanban
* jgart
## Implementation
Key is to leverage what is in Guix and what we already have in GeneNetwork as much as possible.
We should create this separately from genenetwork because all we want to do is run notebooks in containers. We should be able to use and external authentication system, fetch notebooks from git elsewhere and push them on git elsewhere.
binding-lite will be packaged in GNU Guix. GeneNetwork will essentially be a 'rebranding' of binding-lite.
Later we'll add an API to create notebooks on the fly. That is a separate topic.
## Background
The current way of using guix-jupyter is to create a guix environment that contains the guix-jupyter kernel and jupyter.
The user can then run the `jupyter notebook` command which prints a url to the terminal for the user to copy/paste into the browser.
Once the url is opened in the browser, a user has to select the guix kernel and choose the notebook from the file manager menu.
## Initial Design Ideas: guix-notebook-launcher
The first goal is to fire up a jupyter notebook from a local git repository.
Afterwards, this would be implemented with a remote git repository.
We need a command line interface/tool and guile library to automate the process of opening a jupyter notebook from a local git repository.
Some of the tasks that `guix-notebook-launcher` will need to automate is preselecting the guix kernel and opening the browser to the notebook view
of the given url.
Here is an example of how the command line interface would be used:
```
$ guix-notebook-launcher file:///path/to/<git-repository>
To access the notebook, open this file in a browser:
file:///path/to/<git-repository>
or copy and paste one of these URLs:
http://localhost:8890/?token=fc21c17b9ec31eddbea28c06ce086cf71b5605a09610439b
or http://127.0.0.1:8890/?token=fc21c17b9ec31eddbea28c06ce086cf71b5605a09610439b
```
## Managing Jupyter Notebook Dependencies
We'll need a manifest.scm file that will specify the dependencies needed by the guix-jupyter notebook.
Below is an example of the contents of a guix-jupyter notebook repository that can be given to `guix-notebook-launcher`:
```
$ tree <git-repository>
.
├── LICENSE
├── README.md
├── index.ipynb
└── manifest.scm
```
An example of the contents of manifest.scm:
```
(specifications->manifest
'("python-numpy"
"python-pandas"
"python-matplotlib"))
```
## Additional Features and Ideas
One possibility for easing the creation of a manifest.scm in the notebook repository is to parse the requirements.txt/setup.py,
in the case of a python notebook, and generate the manifest.scm file for the notebook user.
We would have to write similar manifest generators for Julia and R project dependencies in order to support this feature.
I will ask on the guix mailing lists for feedback on this idea.
Once we have this library we can start building binder-lite on top of it.
## Use Cases
The use cases for `guix-notebook-launcher` are driven by the following design decisions:
* Automated guix container building guile library to be used by binder-lite.
* Manual guix container building and running the container from the command line,
`guix-notebook-launcher`, by users on their workstations.
## Initial Ideas for binder-lite Front End and Backend
* Single-page application for logging in to a guix-jupyter notebook view.
* Leverage oauth for reusing user registration credentials from other platform websites.
* Git repository url input form for sending to guix-notebook-launcher on the backend.
* Push changes made in notebook session to remote origin repository on third-party git forge.
## Next Development Steps
## Current Notebook Launcher Issues
=> https://paste.sr.ht/~jgart/0c04c5ee583fe8ac7bf69097025e0453592e81e8#jupyter-notebook-fails-to-connect-L14 store-connection-error
=> https://git.sr.ht/~jgart/binderlite-demo/tree/demo1/item/demo.scm#L39 Current Notebook Container Default Settings
## Current WIP Code
=> https://git.sr.ht/~jgart/binderlite-demo/tree/demo1/ binderlite demo
=> https://github.com/jgarte/binderlite-example binderlite example repository
### Script / Command Line Interface / Web UI Mock / GNU Artanis
* Test loading a preconfigured guix-jupyter kernel with `MultiKernelManager.default_kernel_nameUnicode`.
* Fire up a preselected guix kernel from a guile or bash script with $BROWSER.
* Understand what the data is that the UI will be interacting with.
* Start mocking a web UI while continuing to develop guix-notebook-launcher and the data that it will provide to the front-end.
* Finish reading, the source code of binderhub. It is only 10,212 lines of python as of Wed 28 Jul 2021 12:20:10 AM EDT.
* Finish reading the source code of repo2docker. It is only 12,356 lines of python as of Wed 28 Jul 2021 12:21:22 AM EDT.
* Meet with Efraim on Thursday to discuss guix containers and my progress on this project.
* Create a mock similar to notebook.gesis.org with GNU artanis, `sxml->xml` from `(sxml simple)`, or just plain html and vanilla javascript.
* Research production readyness of GNU Artanis and whether we'll need it. Contact Nala Ginrut.
* Launch guix-jupyter session from binder-example git repository similar to binder-example by GeorgianaElena.
### Reference Links
=> https://jupyter-notebook.readthedocs.io/en/latest/config.html Jupyter Notebook Config File
=> https://notebooks.gesis.org/ Gesis Notebooks Front Page
=> https://www.gnu.org/software/artanis/manual/ GNU Artanis
=> https://nalaginrut.com/archives/2021/05/04/scheme%20language%20is%20good%20enough%20for%20products Scheme Language in Production
=> https://github.com/GeorgianaElena/binder-example Binder Example
## Current Questions I Have to Answer
* Is OAuth still needed?
Yes
* What type does a manifest return?
A manifest! Which of course is just another record :)
```
scheme@(guix-user)> ,use (gnu packages) (guix profiles)
scheme@(guix-user)> (manifest? (specifications->manifest '("python-ipython")))
$1 = #t
```
* Pushing? Pulling? Authentification? Via the github API?
* Will we build binder-lite completely from scratch or are there useful pieces from binderhub that we can salvage and while stripping out the docker integration?
* How will we parse the new dependencies that a user introduces during a notebook session?
* Will guix-jupyter magics be available to the notebook user? How will they interact with guix-notebook-launcher data?
* Will dependencies be written back to requirements.scm to update the state at the end of a notebook session? Is this already implemented in Binderhub?
### Reference Links
=> https://hpc.guix.info/blog/2019/10/towards-reproducible-jupyter-notebooks/ Guix Jupyter Magics
## Misc Binder Lite Ideas
Should binder-lite be a distribution of GNU Guix System?
GeneNetwork can provide custom ISO and QEMU images that contain binder-lite ready to be deployed to bare metal or a VM.
This is inspired by the-littlest-jupyterhub distribution.
### Reference Links
=> https://tljh.jupyter.org/en/latest/ The Littlest JupyterHub
## Basic Working Demo
=> https://git.sr.ht/~jgart/binderlite Demo
### TODOs
* Add tests
* Add cloned repository to a uuid generated path so that app doesn't fail on reclone due to previously existing git repo
* Updgrade Flask to 2.0 in GNU Guix.
* Add favicon.
* Add css styling and html meta headers.
* Add Guix logo loading css animation similar to https://github.com/jupyterhub/binderhub/blob/master/binderhub/static/loading.css
* Add xtermjs component while waiting for container to build.
* Cull container after a period of inactivity.
|
