I wouldn’t assume this is done with malice in mind, but maybe this is someone unaware of the importance of a formal license.

Indeed. They only have in their github page:

Terms of Use

Feel free to use these components in personal and commercial projects. However, while the tutorials and demos are available for your use as-is, they cannot be redistributed or resold. Let’s keep things fair and respect each other’s work.

The whole talk is available here: https://www.youtube.com/watch?v=ZNK4aSv-krI

This specific one is at 39min.

I know what you’re talking about, happens to us all when we’re learning something new.

Want to share the details of a specific issue you’re facing, blocking you?

I understand your position. There is a learning curve to containers, but I can assure you that getting your basics on the topic will open a whole new world of possibilities and also make everything much easier for yourself. The vast majority of people run containers which make the services less brittle because they have their own tailored environment and don’t depend on the host libraries and packages and also brings increased security because the services can’t easily escape their boundaries rendering their potential vulnerabilities less of an issue compared to running those same services bare metal.

I started on synology too. There is a website called Marius hosting which focuses on tutorials for containers on synology, but his instructions have been updated the last few years to focus on spinning up containers manually rather than through the UI, which makes it more intimidating than it needs to be for beginners… I’ll link it here just as a reference. I’ll see if on the way back machine he shows the easier way and report back if I find something.

Edit: yes here is an original tutorial for Jellyfin (this method still works for me and is still how I use docker lately): https://web.archive.org/web/20210305002024/https://mariushosting.com/how-to-install-jellyfin-on-your-synology-nas/

To answer your question more specifically, most people set up the pi with docker, using services which have a front end accessible in the browser. They basically use their browser to navigate to the front end of the service they want to use and administer it like that. For instance portainer to manage their docker containers, or pihole for managing their firewall, or even jellyfin for their media which is both the website to consume the media and has an administrator dashboard.

Tailscale is a good option.

Edit: I’m assuming you mean away from home, but if you mean in your local network just use SSH?

I’m very grateful for your extended help. I’ve made some progress. I’m able to get the prompt to appear asking me for my passphrase to unlock the right partition (sda3 in my case). Entering the passphrase, however, drops me in the Dracut emergency shell after ~3min of dracut logs, seemingly looping. (Edit: the reason for why it drops me in the shell is very unclear. It says Dropping to debug shell. /bin/sh: can't access tty: job control turned off. And if I try to exit the dracut shell, it says dracut Warning: could not boot.).

In the Dracut emergency shell, checking /dev/mapper/ I see a luks-<sda3-uuid> listed. Running blkid I see it listed too with TYPE=crypto_LUKS. I also see a dev/dm-0 with a dedicated UUID, in ext4. I ran blkid which shows:

/dev/mapper/luks-705fc477-573a-4ef6-81b6-a14c43cda1f5: UUID="57955343-922a-4918-9bc1-797ca8d13a9c" TYPE="ext4"
/dev/sda1: UUID="cc5e0b03-3544-4bef-ab8b-8b72dd236926" TYPE="ext4"
/dev/sda2: UUID="4df1af6c-3199-4bb2-bb12-bcf897cfc6fc" TYPE="swap"
/dev/sda3: UUID="705fc477-573a-4ef6-81b6-a14c43cda1f5" TYPE="crypto_LUKS"
/dev/dm-0: UUID="57955343-922a-4918-9bc1-797ca8d13a9c" TYPE="ext4"

I checked the status of the filesystem running cryptsetup status /dev/mapper/luks-<sda3-uuid> and it says it is active, which I guess means it is unlocked?

I checked the /root directory, and it is empty. So I tried to mount the partition myself: mount /dev/mapper/luks-<sda3-uuid> /root but it fails saying mount: mounting /dev/mapper/luks-<sda3-uuid> on /root failed: No such file or directory and that got me really puzzled? I’ve been searching far and wide but I can’t seem to find anyone with a similar situation. I feel like I’m close to getting this working.

Below is my syslinux kernel config, and the 2nd and 3rd items are what I booted into (/boot/extlinux.conf)

# Generated by update-extlinux 6.04_pre1-r15
DEFAULT menu.c32
PROMPT 0
MENU TITLE Alpine/Linux Boot Menu
MENU HIDDEN
MENU AUTOBOOT Alpine will be booted automatically in # seconds.
TIMEOUT 10
LABEL lts
  MENU DEFAULT
  MENU LABEL Linux lts
  LINUX vmlinuz-lts
  INITRD initramfs-lts
  APPEND root=/dev/mapper/root modules=sd-mod,usb-storage,ext4 cryptroot=UUID=705fc477-573a-4ef6-81b6-a14c43cda1f5 cryptdm=root rootfstype=ext4 rd.debug log_buf_len=1M rd.shell

LABEL lts
  MENU DEFAULT
  MENU LABEL Dracut Linux lts
  LINUX vmlinuz-lts
  INITRD /boot/initramfs-6.6.56-0-lts.img
  APPEND root=/dev/mapper/luks-705fc477-573a-4ef6-81b6-a14c43cda1f5 modules=sd-mod,usb-storage,ext4 rootfstype=ext4 rd.shell rd.debug log_buf_len=1M rd.luks.uuid=705fc477-573a-4ef6-81b6-a14c43cda1f5

LABEL lts
  MENU DEFAULT
  MENU LABEL Dracut Linux lts 2
  LINUX vmlinuz-lts
  INITRD /boot/initramfs-6.6.56-0-lts.img
  APPEND modules=sd-mod,usb-storage,ext4,dm,crypt,rootfs-block rootfstype=ext4 rootflags=rw,relatime rd.shell rd.debug log_buf_len=1M root=UUID=57955343-922a-4918-9bc1-797ca8d13a9c rd.luks.uuid=705fc477-573a-4ef6-81b6-a14c43cda1f5

And here the /proc/cmdline of the booted partition:

BOOT_IMAGE=vmlinuz-lts modules=sd-mod,usb-storage,ext4,dm,crypt,rootfs-block rootfstype=ext4 rootflags=rw,relatime rd.shell rd.debug log_buf_len=1M root=UUID=57955343-922a-4918-9bc1-797ca8d13a9c rd.luks.uuid=705fc477-573a-4ef6-81b6-a14c43cda1f5 initrd=/boot/initramfs-6.6.56-0-lts.img

Here is my setup, when I boot in my regular initramfs (the one I’m trying to replicate using dracut):

mytestalpine:~# lsblk -o NAME,FSTYPE,FSVER,LABEL,UUID,FSAVAIL,FSUSE%,MOUNTPOINTS
NAME     FSTYPE      FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda                                                                                  
├─sda1   ext4                    cc5e0b03-3544-4bef-ab8b-8b72dd236926  195.5M    21% /boot
├─sda2   swap                    4df1af6c-3199-4bb2-bb12-bcf897cfc6fc                [SWAP]
└─sda3   crypto_LUKS             705fc477-573a-4ef6-81b6-a14c43cda1f5                
  └─root ext4                    57955343-922a-4918-9bc1-797ca8d13a9c    2.3G     8% /

mytestalpine:~# lsblk -l -n /dev/sda3
sda3   8:3   0  2.8G 0 part  
root 253:0   0  2.8G 0 crypt /

Note: No idea of the relevance, but I’m testing this setup in a VM, with a BIOS firmware.

sleep.

my tip: put your phone on silent and place it in another room to prevent you from reflexively taking it while you struggle to fall asleep. just make sure to have the alarm volume be high enough for you to wake up.

Thank you for your help. I spent time digging into this rabbit hole, and while I’ve learned a lot, I am struggling to get the basics to work. Right now, I’m focusing on being able to just boot an image I created using dracut, excluding all the initial stuff I wanted, just be able to reproduce the original functionality of being able to unlock my luks partition using my keyboard.

Where I’m at: I am building my initramfs using the following command: dracut -f -v --add crypt --add lvm --add dm. I get the following output log:

Then I updated the /boot/extlinux.conf file, adding the following second entry (displaying the first one just for comparison):

LABEL lts
  MENU DEFAULT
  MENU LABEL Linux lts
  LINUX vmlinuz-lts
  INITRD initramfs-lts
  APPEND root=/dev/mapper/root modules=sd-mod,usb-storage,ext4 cryptroot=<my-uuid> cryptdm=root quiet rootfstype=ext4

LABEL lts
  MENU LABEL dracut-img
  LINUX vmlinuz-lts
  INITRD /boot/initramfs-6.6.56-0-lts.img
  APPEND root=/dev/mapper/root modules=sd-mod,usb-storage,ext4 cryptroot=UUID=<my-uuid> cryptdm=root quiet rootfstype=ext4 rootflags=rw,relatime

I added the rootflags=rw,relatime because this was shown in the dracut log, so I thought perhaps that mattered. But for the most part I left it the same as the previous entry, because I’m trying to do the same thing I suppose. Perhaps I’m mistaken?

The current result of booting that image leads to a long loading (not asking for the passphrase to unlock the partition) then displaying the following error:

dracut Warning: Could not boot.

dracut Warning: "/dev/mapper/root" does not exist

Generating "/run/initramfs/rdsosreport.txt"
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot after mounting them and attach it to a bug report.

To get more debug information in the report, reboot with "rd.debug" added to the kernel command line.

Dropping to debug shell.

Before dropping me in a shell, in which I have not found anything useful to do. I am surely missing something basic as my understanding of what’s happening is pretty superfluous.

What I’m noticing which may be of importance:

• dracut[E]: ldconfig exited ungracefully, in the dracut output log. Perhaps this matters and should be fixed? An image is nonetheless generated.
• there are many missing modules when creating an image, but I don’t know if any of them matter, at least for my purpose.
• One thing I can’t wrap my head around is, how come the original kernal image work, when I had packages such as device-mapper and lvm missing, why did dracut complain about them missing for me to compile my own image? and would I need to add options in the /boot/extlinux.conf file, when they are not required for the original boot entry, when all I’m trying to do (as a start) is just make sure I can reproduce a bootable kernel image?

Indeed, quite surprising. You got to “stroke their fur the right way” so to speak haha

Also, I’m increasingly more impressed with the rapid progress reaching open-weights models: initially I was playing with Llama3.1-8B which is already quite useful for simple querries. Then lately I’ve been trying out Mistral-Nemo (12B) and Mistrall-Small (22B) and they are quite much more capable. I have a 12GB GPU and so far those are the most powerful models I can run decently. I’m using them to help me in writing tasks for ansible, learning the inner workings of the Linux kernel and some bootloader stuff. I find them quite helpful!

Someone recently referred me to this blog post about using RAG in open-webui. I have not tested if but the author seems to reach a good setup.

https://medium.com/@kelvincampelo/how-ive-optimized-document-interactions-with-open-webui-and-rag-a-comprehensive-guide-65d1221729eb

Perhaps this is of use to you?

Have you installed google services on your phone? they are available through the grapheneOS official “App Store” app. This should be installed before whatsapp is installed (at least that is the instruction for general apps depending on google services).

Perhaps you have done so already, but just a general advice: when using google services and invasive apps like WhatsApp, it can be a good idea to install in their dedicated profile and allow the notifications to pipe through to your main profile instead of installing both in your main profile. If you need help configuring it, let me know.

I have no idea if ollama can handle multi-GPU. The 70B in it’s q2_k quantized form requires already 26GB of memory, so you would need at least that to run it well and that would only imply it could be entirely run on GPU, which is the best case scenario, but not at what speed.

I know some people with apple silicon who have enough memory to run the 70B model and for them it runs fast enough to be usable. You may be able to find more info about it online.

I wish I could. I have an RTX 3060 12GB, I run mostly llama3.1 8B versions in fp8, at 30-35 tokens/s.

Sure! It can be a bit of a steep learning curve at times but there are heaps of resources online, and LLMs can also be useful, even if it just in pointing you in the direction for further reading. Regardless, you can reach out to me or other great folks from the !localllama@sh.itjust.works or similar AI, ML or related communities!

Enjoy :)

For RAG, there are some tools available in open-webui, which are documented here: https://docs.openwebui.com/tutorials/features/rag They have plans for how to expand and improve it, which they describe here: https://docs.openwebui.com/roadmap#information-retrieval-rag-

For fine-tuning, I think this is (at least for now) out of scope. They focus on inferencing. I think the direction is to eventually help you create/manage your own data which you get from using LLMs using Open-WebUI, but the task of actually fine-tuning is not possible (yet) using either ollama or open-webui.

I have not used the RAG function yet, but besides following the instructions on how to set it up, your experience with RAG may also be somewhat limited depending on which embedding model you use. You may have to go and look for a good model (which is probably both small and efficient to re-scan your documents yet powerful to generate meaningful embeddings). Also, in case you didn’t know, the embeddings you generate are specific to an embedding model, so if you change that model you’ll have to rescan your whole documents library.

The interface called open-webui can run in a container, but ollama runs as a service on your system, from my understanding.

The models are local and only answer queries by default. It all happens on the system without any additional tools. Now, if you want to give them internet access, you can, it is an option you have to setup and open-webui makes that possible though I have not tried it myself. I just see it.

I have never heard of any llm “answer base queries offline before contacting their provider for support”. It’s almost impossible for the LLM to do it by itself without you setting things up for it that way.

whats great is that with ollama and webui, you can as easily run it all on one computer locally using the open-webui pip package or in a remote server using the container version of open-webui.

Ive run both and the webui is really well done. It offers a number of advanced options, like the system prompt but also memory features, documents for RAG and even a built in python ide for when you want to execute python functions. You can even enable web browsing for your model.

I’m personally very pleased with open-webui and ollama and they both work wonders together. Hoghly recommend it! And the latest llama3.1 (in 8 and 70B variants) and llama3.2 (in 1 and 3B variants) work very well, even on CPU only, for the latter! Give it a shot, it is so easy to set up :)

The framework laptop, a modular laptop, now has a risc v motherboard, to be used in their computers. Framework prides itself in being a good open source steward and you can read more about the motherboard here and buy it here (when it will be available):

https://frame.work/products/deep-computing-risc-v-mainboard

https://frame.work/blog/introducing-a-new-risc-v-mainboard-from-deepcomputing