DATE: Sun 12 Jan 2025 15:51 By: konsthol@pm.me

Simple way to extend yt-dlp

Lots of people use yt-dlp either directly or indirectly through mpv. It’s a powerful tool that acts as a website scraper and it supports thousands of websites. The website its mostly used for is like the name suggests YouTube. Now, YouTube is a great resource but usage through the website is quite unpleasant so lots of people opt out to use alternative frontends like Invidious or Piped. Lots of times you just want to use mpv to stream a YouTube video by providing the link like:

mpv https://youtube.com/watch?v=[VideoID]

That works like a charm, but what happens when you provide a link of an alternative frontend? Well, it translates it to the aforementioned format in order to work. But there are so many instances of Invidious and Piped, so how does it know what to do? That was my question as well since I use a self hosted Piped instance and it does not recognize the domain. Obviously.

Thankfully, yt-dlp is an open source project so you can actually see what goes on behind the scenes. In my case, I installed it with the Arch Linux package manager and it resides at:

/usr/lib/python3.13/site-packages/yt_dlp/

The way yt-dlp works is that it has a folder called “extractor” in that path and in that folder there is a python file for each supported website. In YouTube’s case it’s youtube.py. I opened it and I saw this:

class YoutubeBaseInfoExtractor(InfoExtractor):
    """Provide base functions for Youtube extractors"""

    _RESERVED_NAMES = (
        r'channel|c|user|playlist|watch|w|v|embed|e|live|watch_popup|clip|'
        r'shorts|movies|results|search|shared|hashtag|trending|explore|feed|feeds|'
        r'browse|oembed|get_video_info|iframe_api|s/player|source|'
        r'storefront|oops|index|account|t/terms|about|upload|signin|logout')

    _PLAYLIST_ID_RE = r'(?:(?:PL|LL|EC|UU|FL|RD|UL|TL|PU|OLAK5uy_)[0-9A-Za-z-_]{10,}|RDMM|WL|LL|LM)'

    # _NETRC_MACHINE = 'youtube'

    # If True it will raise an error if no login info is provided
    _LOGIN_REQUIRED = False

    _INVIDIOUS_SITES = (
        # invidious-redirect websites
        r'(?:www\.)?redirect\.invidious\.io',
        r'(?:(?:www|dev)\.)?invidio\.us',
        # Invidious instances taken from https://github.com/iv-org/documentation/blob/master/docs/instances.md
        r'(?:www\.)?invidious\.pussthecat\.org',
        r'(?:www\.)?invidious\.zee\.li',
        [more instances here]
    )

There is a class called YoutubeBaseInfoExtractor that has an array of instances called _INVIDIOUS_SITES that uses a regex to catch every domain there. Now, I saw at the GitHub page of yt-dlp a lot of people asking the maintainers to add more instances on this list. Theoretically you also can just edit the file and add a domain so that it recognizes that one too. But, in my personal opinion it’s never a good idea to edit upstream files because as the program updates your changes will be overwritten. So I found another way to deal with this.

You see, yt-dlp is not just a command line utility. You can use it as a library to make your own extractors for websites. The way you do that is by creating your own plugins. In my case, I didn’t actually want to make a new extractor but somehow extend an array of an already existing one. Not all extractors use this method but since YouTube does, it would work. So I made this file at this location:

~/.config/yt-dlp/plugins/piped/yt_dlp_plugins/extractor/piped.py

The contents are simple:

from yt_dlp.extractor.youtube import YoutubeBaseInfoExtractor, YoutubeIE

class CustomYoutubeBaseInfoExtractor(YoutubeBaseInfoExtractor):
    _INVIDIOUS_SITES = YoutubeBaseInfoExtractor._INVIDIOUS_SITES + (
        r'(?:www\.)?piped\.konsthol\.eu',
    )

class PipedKonstholYoutubeIE(YoutubeIE, CustomYoutubeBaseInfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?piped\.konsthol\.eu/watch\?v=(?P<id>[0-9A-Za-z_-]{11})'
    IE_NAME = 'piped.konsthol.eu'

We import the class that contains the array we need and the youtube extractor. We make a new class in which we provide the one that has the array. We access the array and add a new regex for our domain. Then we make a new class for the extractor, provide the one we just created and the YouTube extractor class and we tell it to work for urls that look like the one we provided. In that way, this pseudo extractor is being activated when we provide a url that looks like this, it extends the actual YouTube extractor and activates that one, only this time it works for our domain too.

It’s amazing what you can do with open source software just by observing how a program works. Now every time someone needs a new domain for an alternative YouTube frontend added, instead of asking the developers to do that, using this simple solution he/she can just add it to the plugin.

yt-dlp GitHub page

..

DATE: Thu 19 Dec 2024 18:35 By: konsthol@pm.me

The magic of Wake-On-LAN

Years ago, some good friends of mine gifted me a Raspberry Pi 4 with 2GB of RAM for my birthday. It’s hands down the most thoughtful gift I’ve ever received, perfectly matching my hobbies. They were lucky to even find one during the chip shortage! I initially used it as a VPN server with WireGuard and played around with Pi-hole for network-wide ad blocking.

Eventually, I got into cloud computing and started hosting multiple services on a VPS I rent from MVPS. This VPS, with its 4GB of RAM, currently runs several of my daily-use services like Gitea, Searx, NTFY, and more. Although the Raspberry Pi could theoretically handle these services, its 2GB of RAM limited simultaneous operations.

One day, it hit me: why not use the Raspberry Pi for a simpler project like a Wake-On-LAN (WOL) server? It requires minimal resources and just needs to stay on and send WOL packets. The Raspberry Pi is connected via Wi-Fi to the same network as my laptop. Normally, waking up my laptop with WOL would require an Ethernet cable connection to the router, but I connected one end to the Pi and the other to the laptop.

I started with a simple shell script using etherwake, a command-line tool to send WOL packets. After enabling WOL in my laptop’s BIOS and confirming it on the OS using ethtool, I could easily wake my laptop remotely using a Termux shortcut on my phone, which SSHed into the Raspberry Pi to execute the etherwake command.

Though this setup worked perfectly fine, I wanted to make it even better. Initially, the Raspberry Pi only ran SSH and executed a command to wake my laptop. So it wasn’t really a WOL server. After some research, I found Flask and created an actual WOL server. This server had routes, authentication, logging, rate limits, and more to ensure robustness and security.

What makes my Flask-based server so cool is its dynamic nature. I can configure a .env file with multiple device MAC addresses, allowing numerous routes for different devices. For instance, I have LAPTOP_MAC=“itsmac” and DESKTOP_MAC=“itsmac”, enabling me to wake them via HTTP requests at /wol/laptop and /wol/desktop, respectively.

One open-source application that fits my use case is HTTP Shortcuts from F-Droid. After configuring a specific route, I can turn it into a widget on my home screen. This way, I can wake my laptop up with just a tap! Plus, I use dynamic DNS, so my Pi is accessible from anywhere.

The project’s mascot

I would suggest anyone to take a look at my project’s repo and if they find that it fits their needs, use the setupSingleBinary.sh script to grab the latest executable. The repo is over at

WOL-Ly

..

DATE: Sat 25 Feb 2023 22:42 By: konsthol@pm.me

WebCall

One of the simplest ways to host a “telephony” service is to use webcall.

Like with emails, you can use any webcall server to communicate with any other webcall server. Meaning that if for example you register a number with https://timur.mobi/ which is the default public server you can still make a call to someone who has registered a number with https://talk.konsthol.eu if you know their number by replacing the server address field in the dialpad icon.

All calls are encrypted and the server does not have access to the content of the calls. That is because the server uses webrtc and tries to make peer to peer connections.

There is also an integrated TURN server which is used if the peers can’t connect directly but if the connection is made though that then no video transmission is possible. I suspect that this is a limitation of the TURN server used in webcall because in jitsi meet which uses coturn instead of pion, it is always possible.

Another differentiation of the two services is that in webcall no more than 2 people can join a conversation, which is more similar to how a regular telephone call behaves.

Also there is currently no functionality to block a number as far as I can see but it should be fairly easy to figure out the public ip address and block it like that.

You should give it a shot and see if it works for you. The source code is at a link over at

https://timur.mobi

..

DATE: Wed 17 Aug 2022 00:09 By: konsthol@pm.me

wpgtk is just more convenient

Pywal is a simple KISS principled python application that extracts colors from an image and creates a color pallete to theme different programs

By default there are some templates that are stored in its .cache directory and there is an option to create new templates in its .config directory.

I found it to be simpler to create every template with wpgtk and store it in its templates folder in .config because there it also utilizes symlinks to its respective files and thus you can have every program’s file to manage in one place.

Moreover wpgtk conveniently stores each created colorscheme in a schemes folder in .config and along with that every wallpaper used as a symlink to the actual wallpaper in a wallpapers folder.

It also has a gui interface but I never really found a use for it as one can easily perform every needed action using just the cli.

After creating a script to be run after every use of the program to theme application that need third party scripts like telegram and qutebrowser the process of switching colorschemes on the fly becomes easy as pie and fast as fuck 🤫

Just give it a shot! Maybe I’ll upload my postWPG script in my gitea instance sometime soon and create an actual tutorial on how to use wpgtk

..

DATE: Sun 12 Sep 2021 00:53 By: konsthol@pm.me

Awesome file sharing tool

Croc is one of the best easy to use file sharing cli tools I came across and only recently discovered how easy it can be to self host your own relay.

Croc as far as I understand needs an intermediary to staple as it says the connection between two computers. If I grasp the concept correctly if two computers can both connect to a third one then a connection between them can be established. The files sent over don’t get first uploaded to the third one and then downloaded to the second but somehow get uploaded and downloaded simultaneously.

I’ll have to further read about that but in short if you have a computer like a server with a static ip that is easily accessible from the internet then because of the great job the author did you can host a relay simply by typing the command

croc relay

which will by default use the ports 9009-9013 but you can specify your own with the –ports arguement like

croc relay –ports 1111,1112

Then if you make that into a systemd service (which I won’t be covering just yet cause I don’t really understand them and only make them by copy pasting basic templates 😐) it’s basically a set and forget kind of thing and you can now send files to anyone in any operating system by using croc by typing

croc –relay “your.domain:9009” send [file]

and the recipient can download it by typing

croc –relay “your.domain:9009” [code]

This is super useful cause there was a time when the public instance was down and I couldn’t send files when it was just so easy to not be dependent on the public instance in the first place.

You should definetely read more about it on Croc’s github page and set up your own relay asap 😎

https://github.com/schollz/croc

..