This is my first time at the congress, after many years of wanting to go, I finally made it wuhuu! Let’s start with the fact that it’s an overwhelming experience, I haven’t had so many new impressions and exciting things happening in such a short time in a long time. For me, the congress felt like a mix between a conference and a festival, the many gatherings and open spaces filled with everything you can imagine, plus the curated talks and self-organized session that left a lasting impression on me and allowed me to learn a lot, so here I want to share some of the talks I listened to. This is a random collection and there was/is still much more to explore, but this might give you some inspiration to start with.

First up is: “Cosmic Connectivity - Starlink, Satellite Swarms and the Hackers’ Final Frontier” - this talk was about LEO satellites that provide connectivity services like Starlink, besides the facts about the constellation of these small satellites, how they are stacked in multiple layers and the rockets used to transport them, the economic aspects were especially interesting. The presenter talked about the huge growth and valuation of Starlink and the huge gap for payload transportation as sending satellites into space becomes more popular. I guess this will make someone even richer if other companies don’t accelerate their development of rockets and space services.

Next I would like to mention the Science Slam on day 2 which was about saving the world with science if people listened to science, the talk contained a lot of memes and its best to see for yourself.

The NGO behind Tor gave an interesting technical talk about Tor, Tor censorship, and how they circumvent it in Russia, Iran, and Turkmenistan. The talk was also a bit amusing as the audience got a glimpse into the Russian censorship department, which apparently works with Excel lists to manually track and block Tor bridges.

Social Engineering, now more important than ever, find out how to protect yourself and your loved ones.

There was a lot to say about what happened in the last four years without a congress. Listen to some of the CCC people talk about it in the CCC Four Year Review.

Synthetic Sentience! The presenter spoke very fast and his topic is complex, it’s probably best if I don’t try to explain it and you watch it yourself! But in short, it is about AGI and very futuristic!

Perhaps my absolute favorite performance, the Heimlich Manover. A super interesting talk about your right to make inquiries to the German public administration and the incredible platform that has been built to help you do that: fragdenstaat.de. The talk also included many cases in connection to the platform.

Do you know what 4D cubes look like? Well, neither do I, but I got more confused and had a lot of fun with “Exploring hypercomputation with the effective topos”. I loved the thought experiment where you imagine a 2D world with 2D beings, one of them is in prison, and now you as a 3D being visit this world, pick up the 2D being and put it on the other side of the prison, it was pretty wild to say the least.. but certainly one of my favorites, and it is a shame that I was not able to see more of the “Wondrous Mathematics” series.

Last event I watched before my departure today was about the things you need to do to actually block your lost or stolen Girocard in Germany, you might think it is enough to call a number, but this talk revealed that you also need to go to the police to trigger the KUNO system, otherwise attackers might still be able to use your card with a signature.

What else? Well, I’m still catching up as there’s so much going on at the same time, but here are some ideas of things I’ll be watching the recordings of, some of which we couldn’t get into as the halls were completely full, so they must be promising!

• Hirne hacken: Hackback Edition

• How to build a submarine and survive (see the submarine exhibition below)

• Back in the Driver’s Seat: Recovering Critical Data from Tesla Autopilot Using Voltage Glitching
• ..and many more, check out media.ccc.de

On day 3, I loved the entertainment that took place at midnight that day: The Prompt Battle. In this battle, which was hosted by two awesome moderators who made it really fun to watch, the contestants had to enter prompts for DALLE to create the best image given a scenario. This was then judged by the audience, here is an example from last night.

Sadly this was not recorded and was only enjoyable in the moment and what better way to end this blog post if not by a DALLE 2 generated picture of my impression prompt of the congress this year.

And with that, I’m calling it a year and I wish you a fantastic new year and hope to see you at 38C3!

This post may get some updates in the future- so come back later

When things get complex, it is not feasible to have only one image for all users. JupyterHub allows you to specify profiles with different images that a user can choose from when logging in. Check the Customizing User Environment section in the Jupyterhub documentation for that. Here is an example that uses an additional ConfigMap in K8s to store profile information:

apiVersion: v1
kind: ConfigMap
metadata:
  name: jhub-profiles
  namespace: jhub
data:
  values.yaml: |
    singleuser:
      profileList:
        - display_name: "Default environment"
          description: "some description"
          default: True
        - display_name: "Default environment - python 3.8"
          description: "Same environment as the default but ..."
          kubespawner_override:
            image: <custom-image>

You can the use e.g. Flux to merge the above when applied:

  valuesFrom:
    - kind: ConfigMap
      name: jhub-profiles
      valuesKey: values.yaml

One approach to building these images is in layers, starting with a base layer that contains only a base OS image and some configurations and installations that apply to all other images, for example, ca-certificates or tools like wget.

So if you already have an image that contains everything, it might be a good idea to reverse engineer it into several layers of images that you can then stack on top of each other to create specific user images without dependency conflicts. Look at how the Jupyter community itself manages its images, or this project I’ve been personally involved in. Both use GitHub Actions to build and publish their images and in the second project I linked, we also involved users on GitHub to suggest and implement features in the images.

Depending on how complex you want to get with this process, you can also work with variables, where you specify different versions or packages when building the images. Jinja is also useful in this context as a templating engine.

Today I learned a neat trick: if your institution doesn’t support VPN, you can still access internal URLs using an SSH proxy. Here are the steps:

1. Open a terminal and run ssh -ND 9999 user@hostname.domain.
2. In Firefox, change the SOCKS settings to localhost, port 9999 and SOCKS v5.

Voila! No VPN? No problem. Access internal URLs easily with this SSH proxy workaround.

Here’s the YAML action configuration:

name: Workflow Usage Datadog Custom Metric

author: Domenic Gosein

description: GitHub Action to send minutes usage per workflow to a custom metric in Datadog.

branding:
  icon: 'clock'  
  color: 'orange'

inputs:
  api_key:
    description: 'Datadog API key e.g. from GitHub Actions Secrets'
    required: true
  app_key:
    description: 'Datadog Application Key e.g. from GitHub Actions Secrets.'
    required: true
  repo_path:
    description: 'Repository path e.g. from standard variable.'
    required: true
  wf:
    description: 'Workflow name from strategy matrix.'
    required: true
  tag_workflow_id:
    description: 'Workflow name from strategy matrix.'
    required: true
  github_tk:
    description: 'Github Actions Token from GitHub Actions Secrets.'
    required: true

runs:
  using: 'composite'
  steps:
    - name: Get latest timing trough GitHub API request
      uses: octokit/request-action@v2.1.0
      id: get_latest_timing
      with:
        route: GET /repos/$/actions/workflows/$/timing
      env:
        GITHUB_TOKEN: $

    - name: Value check received from API request
      shell: bash
      run: echo $

    - name: Calculate and post latest timing in minutes to Datatdog as a Custom Metric
      shell: bash
      id: post_latest_timing
      run: |
          min=$(($latest_total_ms_ubuntu / 60000))
          echo $min
          export NOW="$(date +%s)"
          curl -X POST "https://api.datadoghq.eu/api/v1/series" \
          -H "Content-Type: text/json" \
          -H "DD-API-KEY: $DD_API_KEY" \
          -d @- << EOF
          {
            "series": [
              {
                "metric": "github.actions.minutes",
                "points": [[${NOW}, $min]],
                "tags": "workflow:$tag_workflow_id",
                "type": "gauge"
              }
            ]
          }
          EOF
      env:
        DD_API_KEY: $
        DD_APP_KEY: $
        latest_total_ms_ubuntu: $
        tag_workflow_id: $

And here’s an exemplary configuration where workflows are defined in a matrix:

on:
  workflow_run:
    workflows: [Worfklow One, Workflow Two] # Runs when one of the workflow defined in the brackets is run and completed
    types:
      - completed
  schedule:
    - cron: 30 23 * * 1,3,5 # At 23:30 on Monday, Wednesday, and Friday
  workflow_dispatch:

jobs:
  updateWorkflowTimings:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        WFs: [workflow_one.yml, workflow_two.yml] # Runs for each workflow defined in the brackets
    steps:
      - name: Use my action
        uses: goseind/datadog-gh-actions@v1.0.0
        with:
          api_key: $
          app_key: $
          repo_path: $
          tag_workflow_id: $
          wf: $
          github_tk: $

This is how it would manifest in a Datadog Notebook with a neat visualization of the GitHub actions minutes:

In the intricate web of environmental conservation and technological innovation, ReCreate emerges as a comprehensive system composed of three pivotal components: ReNate, SeDrone, and VeDA. This collective ensemble works synergistically to investigate and evaluate abandoned landscapes, formulating a tailored renaturation strategy precisely attuned to each unique terrain.

At its core, ReCreate embraces a blend of autonomy and human involvement, epitomizing a harmonious partnership between cutting-edge technology and human intellect. Let’s delve into the key constituents of this transformative system:

1. ReNate: This element serves as the guiding intelligence, orchestrating the initial reconnaissance of deserted lands. ReNate’s role extends beyond mere data collection; it analyzes environmental parameters, identifies ecological imbalances, and lays the foundation for a targeted renaturation approach.

2. SeDrone: Taking to the skies, SeDrone is the aerial counterpart that complements ReNate’s ground-level insights. Equipped with advanced sensors, SeDrone provides a bird’s-eye perspective, capturing critical data to enhance the overall understanding of the landscape. This synergy of ground and aerial perspectives ensures a comprehensive assessment.

3. VeDA: As the brainchild of ReCreate, VeDA synthesizes the amassed data and formulates a renaturation strategy tailored to the specific needs of the identified area. This analytical engine leverages machine learning algorithms, ecological models, and historical data to predict and optimize the restoration process.

While ReCreate operates as a partially autonomous system, it is imperative to recognize the indispensable role of human expertise in its functionality. Humans play a pivotal part in the planning and execution phases, bringing a nuanced understanding of local ecosystems, cultural considerations, and ethical dimensions. The collaborative effort between technology and human insight amplifies the effectiveness and sustainability of the renaturation initiatives.

In essence, ReCreate exemplifies a holistic approach to environmental stewardship, seamlessly integrating the precision of technology with the wisdom of human experience. As we navigate the complex terrain of ecological restoration, ReCreate stands as a beacon, demonstrating that the future of sustainable ecosystems lies in the convergence of nature and cutting-edge innovation.

This video illustrates the system in operation.

Discover additional details about the project here!

Today I learnt about watchdog, a Python utility that allows you to monitor file system events, useful for example if you want to restart a script when changes are made to it. I used it with Docker Compose to run a Python program every time I changed the file. Create a small bash script to run watchmedo, specifying the directory, pattern or single file you want to watch (in this case using the $1 variable to pass the specific file from the Docker Compose later):

#!/bin/bash
echo "Starting '$1' with auto-restart on file change ..."
watchmedo auto-restart --directory=./ --pattern=*.py --recursive -- python3 $1

Add the file and pip package to your Dockerfile (make sure it has the right permissions):

# install pip packages
RUN pip install watchdog
# setup scripts and entrypoint
COPY ./watchdog.sh /
RUN chmod +x watchdog.sh

In your Docker Compose, you can now run the script and specify a file to track:

services:
  svc:
    build: .
    command: /watchdog.sh test.py # replace this with any file you want to update and restart

That’s it!

Expo is a set of tools, libraries, and services built around React Native to simplify the development process. It provides a set of pre-built components and a development environment that eliminates the need for setting up and configuring native build tools. Expo also offers a range of services, such as a cloud build service, which allows you to build and deploy your app without the need for Xcode or Android Studio. Expo is particularly helpful for developers who want to focus on the JavaScript side of their application and reduce the complexity of native development tasks. Now, let’s dive in and build.

The easiest way to get started is with Snack, a convenient online platform provided by Expo that allows developers to experiment with and share React Native code in a browser-based environment including Android and iOS emulators powered by Appetize.

This allows you to draft a simple version of your new app without having to worry about installing and configuring things. Once you have a minimal working version of your app you can move to a more sophisticated setup and start working with EAS CLI tools to build and later submit your app.

With preview APK builds, configured through the eas.json file in your project, you can share and download them directly on any device once finished, you’ll be provided with a QR code.

{
  "cli": {
    "version": ">= 5.6.0"
  },
  "submit": {
    "production": {
      "android": {
        "track": "internal",
        "releaseStatus": "draft"
      }
    }
  },
  "build": {
    "preview": {
      "distribution": "internal",
      "android": {
        "buildType": "apk"
      }
    }
  }
}

With the submitted production profile also listed in the example above you can automatically build ABBs for Google Play Store and submit them there.

This can also be nicely automated with GitHub Actions.

More to follow..

Zooming in on the container diagram shows the individual components of each software system in scope and their interactions across systems.

The cool thing is that C4 is a DSL and you can save all of the above as code, e.g. along with your source code of the systems.

workspace "Ball Schubser" "A robot that pushes a ball" {

    model {
        ballschubser = softwareSystem "Ball Schubser" "Software System for Ball-Schubser" {
            cockpit = container "Cockpit" "Webpage with robots camera and status in real-time" {
            }
            detectnode = container "Object Detection Node" "Real-time object detection with YOLO" {
            }
            navnode = container "Navigation Node" "Publisher for navigation instructions" {
            }
            ros = container "ROS Core" "Roboter Operating System Master" {
            }
        }
        bot = softwareSystem "Turtlebot" "Bot with camera and Raspberry Pi with WiFi module" {
            rosbot = container "ROS Robot Node"
            camera = container "Camera"
            controller = container "Robot Controller"
    }
    camera -> rosbot "raw image"
    rosbot -> ros "raw image"
    ros -> detectnode "raw image"
    detectnode -> ros "image with detections"
    detectnode -> ros "ball position"
    ros -> navnode "ball position"
    ros -> cockpit "mission status"
    ros -> cockpit "image with detections"
    navnode -> ros "instructions"
    ros -> rosbot "instructions"
    rosbot -> controller "instructions"
}

I find this a very practical approach to architecture design, and it has helped me a lot as it provides an easy and fast way to create the first draft but then allows you to build on that and create more sophisticated diagrams and look at them with different views to understand how things interact.

For more information about C4 and the many ways you can use it, click here:

• C4 Model
• Structurizr DSL cookbook
• Language reference

I can also recommend Structurizr, a dedicated platform with an online DSL editor and diagram download and embed.

Project Jupyter and IPython have no doubt revolutionized the way scientific computing is done. Today CERN and many other major organisations around the globe are using them to do their research. This article gives you the newest insights into Project Jupyter from its biggest community gathering in 2023 in Paris - the JupyterCon!

Project Jupyter is a large open-source project born out of the IPython project for interactive data science and scientific computing with an international community. Jupyter products are used by many large institutes and corporations like NYU, MIT, Harvard, Berkeley, NASA, Netflix, AWS, and Google (and of course CERN) for their scientific and educational needs as well as many science experiments from various disciplines across the globe. Since its start in 2014, it has grown bigger by the year and revolutionized the way science is done.

Source: https://jupyter.org/, 05/2023

At JupyterCon this international community comes together once a year to present and discuss their advancements, use cases and products in and around the expanding Jupyter Ecosystem. The experience is enriched by insightful keynotes, in-depth training and networking events. The conference had several tracks ranging from Community Tools and Practices to Enterprise Jupyter Infrastructure, Data Science, Education and Research and Scientific Computing. This year’s conference took place at Cité des sciences et de l’industrie - an amazing venue in Paris - with around 300 attendees.

Throughout the conference, there were many opportunities to network with other attendees, many of whom are facing the same challenges in their domains as we at CERN. Open science, data preservation and reproducibility are pressing topics in science and the conference provided a good opportunity to discuss these with researchers from MIT, NASA, the Memorial University of Newfoundland working with NASA TOPS on open science as well as the Leibniz Institute for Information Infrastructure in Germany. Apart from the above, there were many more fruitful conversations with maintainers, admins and users of the Jupyter Ecosystem.

The conference was packed with interesting talks, here are just some of the key takeaways I found particularly interesting also in the context of CERN: The new major release of the JupyterHub subproject brings with itself:

• Advanced testing against K8s with CI/CD workflows
  • Ready to use Grafana Dashboards for monitoring of Jupyter services
  • Advances build in user management with groups for e. g. resources limitation
  • Real-time collaboration on Jupyter Notebooks
  • pytest-jupyterhub package for lightweight local JupyterHub development
• Paul Romer, a University Professor at NYU and co-recipient of the 2018 Nobel Prize in Economics Sciences talked in his keynote about scientific publications, reproducibility and signing and preservation of Jupyter Notebooks. Link to this talk
• 2i2c is a non-profit organisation which designs, develops, and operates JupyterHubs in the cloud for communities of practice in research & education. It builds and supports open-source infrastructure that serves these communities and has a lot of experience and infrastructure code to discuss and explore. Link to their GitHub Orga as well as their talk
• Alyssa Goodman who is the Robert Wheeler Willson Professor of Applied Astronomy at Harvard University and a Research Associate at the Smithsonian Institution talked in her keynote about glue, a package which combines data, graphs and tools for the context of astronomy and other fields, with an upcoming package for Jupyter called glupyter. Find more info here
• Github presented its integration of Jupyter into GitHub Codespaces as a native environment for users to open notebooks with, Jupyter has its own sponsored service for that called BinderHub, more info can be found here and here
• New JupyterLite, a Jupyter Notebook running entirely in the browser, making it easier to set up and share notebooks and comes with a dedicated repository spawner called repo2jupyterlite. More info can be found here
• Sliver is a hacking tool to explore vulnerabilities in software systems and a talk at the conference gave a brief introduction to it, more details here
• A Dask tutorial for distributed computing can be found here Real-time collaboration for Jupyter users, more info here
• Effective Practices for Community Management from an experienced user experience researcher for sustainable software. Her talk can be found here

These are just an excerpt of the many presentations, lightning talks and poster sessions given, the full schedule and soon the recording should be available on the JupyterCon website here

During the conference I had the chance to present our project, the Virtual Research Environment/Data Lake, a European effort under the EOSC Future project with a team of maintainers here at CERN.

Additionally, development sprints took place where developers and other project members could sit together to solve issues and think about new features and general ideas on how to move the project forward. I attended one of these sprints to help resolve issues and get help and feedback on some of the issues we have in our project. These sprints are super useful and are the perfect place to have in-depth technical discussions and find solutions for problems, but also to connect to the community.

CERN with the EOSC Future project is a big user and beneficiary of the IPython and Jupyter products and is planning to get more involved with the upstream project. In case you are interested in collaborating or simply exploring ideas feel free to reach out to me!