mirror/slsk-batchdl
1
0
Fork 0
mirror of https://github.com/fiso64/slsk-batchdl.git synced 2024-12-22 14:32:40 +00:00
Code Issues Activity

big commit

Browse source
This commit is contained in:
fiso64 2024-08-11 16:53:11 +02:00
parent 939a867754
commit aef9147b1e
21 changed files with 7402 additions and 5266 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

718
README.md
View file

@ -2,281 +2,529 @@
A batch downloader for Soulseek built with Soulseek.NET. Accepts CSV files and Spotify or YouTube urls.
## Index
- [slsk-batchdl](#slsk-batchdl)
- [Options](#options)
- [Input types](#input-types)
- [CSV file](#csv-file)
- [YouTube](#youtube)
- [Spotify](#spotify)
- [Bandcamp](#bandcamp)
- [Search string](#search-string)
- [Download modes](#download-modes)
- [Normal](#normal)
- [Album](#album)
- [Aggregate](#aggregate)
- [Album Aggregate](#album-aggregate)
- [Searching](#searching)
- [File conditions](#file-conditions)
- [Name format](#name-format)
- [Skip existing](#skip-existing)
- [Configuration](#configuration)
- [Examples](#examples-1)
- [Notes](#notes)
## Options
```
Usage: sldl <input> [OPTIONS]
Required Arguments
<input> A url, search string, or path to a local CSV file.
Run --help "input" to view the accepted inputs.
Can also be passed with -i, --input <input>
--user <username> Soulseek username
--pass <password> Soulseek password
```
```
General Options
-p, --path <path> Download directory
-f, --folder <name> Subfolder name. Set to '.' to output directly to --path
--input-type <type> Force set input type, [csv|youtube|spotify|bandcamp|string]
--name-format <format> Name format for downloaded tracks. See --help name-format
-n, --number <maxtracks> Download the first n tracks of a playlist
-o, --offset <offset> Skip a specified number of tracks
-r, --reverse Download tracks in reverse order
-c, --config <path> Set config file location. Set to 'none' to ignore config
--profile <name> Configuration profile to use. See --help "config".
--concurrent-downloads <num> Max concurrent downloads (default: 2)
--m3u <option> Create an m3u8 playlist file in the output directory
'none' (default for single inputs): Do not create
'index' (default): Write a line indexing all downloaded
files, required for skip-not-found or skip-existing=m3u
'all': Write the index and a list of paths and fails
-s, --skip-existing Skip if a track matching file conditions is found in the
output folder or your music library (if provided)
--skip-mode <mode> [name|tag|m3u|name-cond|tag-cond|m3u-cond]
See --help "skip-existing".
--music-dir <path> Specify to also skip downloading tracks found in a music
library. Use with --skip-existing
--skip-not-found Skip searching for tracks that weren't found on Soulseek
during the last run. Fails are read from the m3u file.
--display-mode <option> Changes how searches and downloads are displayed:
'single' (default): Show transfer state and percentage
'double': Transfer state and a large progress bar
'simple': No download bars or changing percentages
--print <option> Print tracks or search results instead of downloading:
'tracks': Print all tracks to be downloaded
'tracks-full': Print extended information about all tracks
'results': Print search results satisfying file conditions
'results-full': Print search results including full paths
--debug Print extra debug info
--listen-port <port> Port for incoming connections (default: 49998)
--on-complete <command> Run a specified command whenever a file is downloaded.
Available placeholders: {path} (local save path), {title},
{artist},{album},{uri},{length},{failure-reason},{state}.
Prepend a state number to only download in specific cases:
1:, 2:, 3:, 4: for the Downloaded, Failed, Exists, and
NotFoundLastTime states respectively.
E.g: '1:<cmd>' will only run the command if the file is
downloaded successfully.
```
```
Searching
--fast-search Begin downloading as soon as a file satisfying the preferred
conditions is found. Higher chance to download wrong files.
--remove-ft Remove 'feat.' and everything after before searching
--no-remove-special-chars Do not remove special characters before searching
--remove-brackets Remove square brackets and their contents before searching
--regex <regex> Remove a regexp from all track titles and artist names.
Optionally specify a replacement regex after a semicolon.
Add 'T:', 'A:' or 'L:' at the start to only apply this to
the track title, artist, or album respectively.
--artist-maybe-wrong Performs an additional search without the artist name.
Useful for sources like SoundCloud where the "artist"
could just be an uploader. Note that when downloading a
YouTube playlist via url, this option is set automatically
on a per-track basis, so it is best kept off in that case.
-d, --desperate Tries harder to find the desired track by searching for the
artist/album/title only, then filtering. (slower search)
--fails-to-downrank <num> Number of fails to downrank a user's uploads (default: 1)
--fails-to-ignore <num> Number of fails to ban/ignore a user's uploads (default: 2)
--yt-dlp Use yt-dlp to download tracks that weren't found on
Soulseek. yt-dlp must be available from the command line.
--yt-dlp-argument <str> The command line arguments when running yt-dlp. Default:
"{id}" -f bestaudio/best -cix -o "{savepath}.%(ext)s"
Available vars are: {id}, {savedir}, {savepath} (w/o ext).
Note that with -x, yt-dlp will download webms in case
ffmpeg is unavailable.
--search-timeout <ms> Max search time in ms (default: 6000)
--max-stale-time <ms> Max download time without progress in ms (default: 50000)
--searches-per-time <num> Max searches per time interval. Higher values may cause
30-minute bans, see --help "search". (default: 34)
--searches-renew-time <sec> Controls how often available searches are replenished.
See --help "search". (default: 220)
```
```
Spotify
--spotify-id <id> spotify client ID
--spotify-secret <secret> spotify client secret
--remove-from-source Remove downloaded tracks from source playlist
```
```
YouTube
--youtube-key <key> Youtube data API key
--get-deleted Attempt to retrieve titles of deleted videos from wayback
machine. Requires yt-dlp.
--deleted-only Only retrieve & download deleted music.
```
```
CSV Files
--artist-col Artist column name
--title-col Track title column name
--album-col Album column name
--length-col Track length column name
--album-track-count-col Album track count column name (sets --album-track-count)
--yt-desc-col Youtube description column (improves --yt-parse)
--yt-id-col Youtube video id column (improves --yt-parse)
--time-format <format> Time format in Length column of the csv file (e.g h:m:s.ms
for durations like 1:04:35.123). Default: s
--yt-parse Enable if the CSV contains YouTube video titles and channel
names; attempt to parse them into title and artist names.
--remove-from-source Remove downloaded tracks from source CSV file
```
```
File Conditions
--format <formats> Accepted file format(s), comma-separated, without periods
--length-tol <sec> Length tolerance in seconds
--min-bitrate <rate> Minimum file bitrate
--max-bitrate <rate> Maximum file bitrate
--min-samplerate <rate> Minimum file sample rate
--max-samplerate <rate> Maximum file sample rate
--min-bitdepth <depth> Minimum bit depth
--max-bitdepth <depth> Maximum bit depth
--banned-users <list> Comma-separated list of users to ignore
--pref-format <formats> Preferred file format(s), comma-separated (default: mp3)
--pref-length-tol <sec> Preferred length tolerance in seconds (default: 3)
--pref-min-bitrate <rate> Preferred minimum bitrate (default: 200)
--pref-max-bitrate <rate> Preferred maximum bitrate (default: 2500)
--pref-min-samplerate <rate> Preferred minimum sample rate
--pref-max-samplerate <rate> Preferred maximum sample rate (default: 48000)
--pref-min-bitdepth <depth> Preferred minimum bit depth
--pref-max-bitdepth <depth> Preferred maximum bit depth
--pref-banned-users <list> Comma-separated list of users to downrank
--strict-conditions Skip files with missing properties instead of accepting by
default; if --min-bitrate is set, ignores any files with
unknown bitrate.
```
```
Album Download
-a, --album Album download mode
-t, --interactive Allows to select the wanted folder and images
--album-track-count <num> Specify the exact number of tracks in the album. Add a + or
- for inequalities, e.g '5+' for five or more tracks.
--album-ignore-fails Do not skip to the next source and do not delete all
successfully downloaded files if one of the files in the
folder fails to download
--album-art <option> Retrieve additional images after downloading the album:
'default': No additional images
'largest': Download from the folder with the largest image
'most': Download from the folder containing the most images
'most-largest': Do most, then largest
--album-art-only Only download album art for the provided album
--no-browse-folder Do not automatically browse user shares to get all files in
in the folder
```
```
-g, --aggregate Aggregate download mode: Find and download all distinct
songs associated with the provided artist, album, or title.
--min-users-aggregate <num> Minimum number of users sharing a track or album for it to
be downloaded in aggregate mode. (Default: 2)
--relax-filtering Slightly relax file filtering in aggregate mode to include
more results
```
```
Help
-h, --help [option] [all|input|download-modes|search|name-format|
file-conditions|skip-existing|config]
```
```
Notes
Acronyms of two- and --three-word-flags are also accepted, e.g. --twf. If the option
contains the word 'max' then the m should be uppercase. 'bitrate', 'sameplerate' and
'bitdepth' should be all treated as two separate words, e.g --Mbr for --max-bitrate.
Flags can be explicitly disabled by setting them to false, e.g '--interactive false'
```
## Input types
The input type is usually determined automatically. To force a specific input type, set
--input-type [spotify|youtube|csv|string|bandcamp]. The following input types are available:
### CSV file
Path to a local CSV file: Use a csv file containing track info of the songs to download.
The names of the columns should be Artist, Title, Album, Length, although alternative names
are usually detected as well. Only the title or album column is required, but extra info may
improve search results. Every row that does not have a title column text will be treated as an
album download.
### YouTube
A playlist url: Download songs from a youtube playlist.
The default method to retrieve playlists doesn't always return all videos, especially not
the ones which are unavailable. To get all video titles, you can use the official API by
providing a key with --youtube-key. Get it here https://console.cloud.google.com. Create a
new project, click "Enable Api" and search for "youtube data", then follow the prompts.
Tip: For playlists containing music videos, it may be better to remove all text in parentheses
(to remove (Lyrics), (Official), etc) and disable song duration checking:
--regex "[\[\(].*?[\]\)]" --pref-length-tol -1
### Spotify
A playlist/album url or 'spotify-likes': Download a spotify playlist, album, or your
liked songs. --spotify-id and --spotify-secret are required in addition when downloading
a private playlist or liked music.
The id and secret can be obtained at https://developer.spotify.com/dashboard/applications.
Create an app and add http://localhost:48721/callback as a redirect url in its settings.
### Bandcamp
An bandcamp url: Download a single track, and album, or an artist's entire discography.
Extracts the artist name, album name and sets --album-track-count=""n+"", where n is the
number of visible tracks on the bandcamp page.
### Search string
Name of the track, album, or artist to search for: Can either be any typical search string
(like what you would enter into the soulseek search bar), or a comma-separated list of
properties like 'title=Song Name, artist=Artist Name, length=215'.
The following properties are allowed:
```
title
artist
album
length (in seconds)
artist-maybe-wrong
```
Example inputs and their interpretations:
```
Input String | Artist | Title | Album | Length
---------------------------------------------------------------------------------
'Foo Bar' (without any hyphens) | | Foo Bar | |
'Foo - Bar' | Foo | Bar | |
'Foo - Bar' (with --album enabled) | Foo | | Bar |
'Artist - Title, length=42' | Artist | Title | | 42
'artist=AR, title=T, album=AL' | AR | T | AL |
```
## Download modes
### Normal
The program will download a single file for every input entry.
### Album
The program will search for the album and download an entire folder including non-audio
files. Activated when the input is a link to a spotify or bandcamp album, when the input
string or csv row has no track title, or when -a/--album is enabled.
### Aggregate
With -g/--aggregate, the program will first perform an ordinary search for the input, then
attempt to group the results into distinct songs and download one of each kind. A common use
case is finding all remixes of a song or printing all songs by an artist that are not your
music dir.
Two files are considered equal if their inferred track title and artist name are equal
(ignoring case and some special characters), and their lengths are within --length-tol of each
other.
Note that this mode is not 100% reliable, which is why --min-users-aggregate is set to 2 by default,
i.e. any song that is shared by only one peer will be ignored. Enable --relax-filtering to
make the file filtering less aggressive.
### Album Aggregate
Activated when --album and --aggregate are enabled, in this mode sldl searches for the query
and groups results into distinct albums. Two folders are considered same if they have the
same number of audio files, and the durations of the files are within --length-tol of each
other (or within 3 seconds if length-tol is not configured). If both folders have exactly one
audio file with similar lengths, also checks if the inferred title and artist name coincide.
More reliable than normal aggregate due to much simpler grouping logic.
Note that --min-users-aggregate is 2 by default, which means that folders shared by only one
peer are ignored.
## Searching
### Soulseek's rate limits
The server will ban you for 30 minutes if too many searches are performed within a short
timespan. The program has a search limiter which can be adjusted with --searches-per-time
and --searches-renew-time (when the limit is reached, the status of the downloads will be
"Waiting"). By default it is configured to allow up to 34 searches every 220 seconds.
The default values were determined through experimentation, so they may be incorrect.
### Quality vs Speed
The following options will make it go faster, but may decrease search result quality or cause
instability:
- --fast-search skips waiting until the search completes and downloads as soon as a file
matching the preferred conditions is found
- --concurrent-downloads - set it to 4 or more
- --max-stale-time is set to 50 seconds by default, so it will wait a long time before giving
up on a file
- --searches-per-time increase at the risk of bans.
### Quality vs Quantity
The options --strict-title, --strict-artist and --strict-album will filter any file that
does not contain the title/artist/album in the filename (ignoring case, bounded by boundary
chars).
Another way to prevent false downloads is to set --length-tol to 3 or less to make it ignore
any songs that differ from the input by more than 3 seconds. However, all 4 options are already
enabled as 'preferred' conditions by default, meaning that such files will only be downloaded
as a last resort anyways. Hence it is only recommended to enable them if you need to minimize
false downloads as much as possible.
## File conditions
Files not satisfying the required conditions will not be downloaded. Files satisfying pref-
conditions will be preferred; setting --pref-format "flac,wav" will make it download lossless
files if available, and only download lossy files if there's nothing else.
There are no default required conditions. The default preferred conditions are:
```
format = mp3
length-tol = 3
min-bitrate = 200
max-bitrate = 2500
max-samplerate = 48000
strict-title = true
strict-album = true
accept-no-length = false
```
sldl will therefore prefer mp3 files with bitrate between 200 and 2500 kbps, and whose length
differs from the supplied length by no more than 3 seconds. It will also prefer files whose
paths contain the supplied artist and album (ignoring case, and bounded by boundary characters)
and which have a non-null length. Changing the last three preferred conditions is not recommended.
### Important note
Some info may be unavailable depending on the client used by the peer. For example, the standard
Soulseek client does not share the file bitrate. If (e.g) --min-bitrate is set, then sldl will
still accept any file with unknown bitrate. You can configure it to reject all files where one
or more of the checked properties is null (unknown) by enabling --strict-conditions.
As a consequence, if --min-bitrate is also set then any files shared by users with the default
client will be ignored. Also note that the default preferred conditions will already affect
ranking with this option due to the bitrate and samplerate checks.
Conditions can also be supplied as a semicolon-delimited string with --cond and --pref, e.g
--cond "br>=320;f=mp3,ogg;sr<96000"
## Name format
Variables enclosed in {} will be replaced by the corresponding file tag value.
Name format supports subdirectories as well as conditional expressions like {tag1|tag2} - If
tag1 is null, use tag2. String literals enclosed in parentheses are ignored in the null check.
### Examples:
- "{artist} - {title}"
Always name it 'Artist - Title'. Because some files on Soulseek are untagged, the
following is generally preferred:
- "{artist( - )title|filename}"
If artist and title are not null, name it 'Artist - Title', otherwise use the original
filename.
### Available variables:
```
artist First artist (from the file tags)
sartist Source artist (as on CSV/Spotify/YouTube/etc)
artists Artists, joined with '&'
albumartist First album artist
albumartists Album artists, joined with '&'
title Track title
stitle Source track title
album Album name
salbum Source album name
year Track year or date
track Track number
disc Disc number
filename Soulseek filename without extension
foldername Default sldl folder name
extractor Name of the extractor used (CSV/Spotify/YouTube/etc)
```
## Skip existing
sldl can skip files that exist in the download directory or a specified directory configured with
--music-dir.
The following modes are available for --skip-mode:
### m3u
Default when checking in the output directory.
Checks whether the output m3u file contains the track in the '#SLDL' line. Does not check if
the audio file exists or satisfies the file conditions (use m3u-cond for that).
### name
Compares filenames to the track title and artist name to determine if a track already exists.
Specifically, a track will be skipped if there exists a file whose name contains the title
and whose full path contains the artist name.
### tag
Compares file tags to the track title and artist name. A track is skipped if there is a file
whose artist tag contains the track artist and whose title tag equals the track title
(ignoring case and ws). Slower than name mode as it needs to read all file tags.
### m3u-cond, name-cond, tag-cond
Default for checking in --music-dir: name-cond.
Same as the above modes but also checks whether the found file satisfies necessary conditions.
Equivalent to the above modes if no necessary conditions have been specified (except m3u-cond
which always checks if the file exists). May be slower and use a lot of memory for large
libraries.
## Configuration
### Config Location:
sldl will look for a file named sldl.conf in the following locations:
```
~/AppData/Roaming/sldl/sldl.conf
~/.config/sldl/sldl.conf
```
as well as in the directory of the executable.
### Syntax:
Example config file:
```
username = your-username
password = your-password
pref-format = flac
fast-search = true
```
Lines starting with hashtags (#) will be ignored. Tildes in paths are expanded as the user
directory.
### Configuration profiles:
Profiles are supported:
```
[lossless]
pref-format = flac,wav
```
To activate the above profile, run --profile "lossless". To list all available profiles,
run --profile "help".
Profiles can be activated automatically based on a few simple conditions:
```
[no-stale]
profile-cond = interactive && download-mode == "album"
max-stale-time = 999999
# album downloads will never be automatically cancelled in interactive mode
[youtube]
profile-cond = input-type == "youtube"
path = ~/downloads/sldl-youtube
# download to another location for youtube
```
The following operators are supported: &&, ||, ==, !=, ! (negation for bools).
The following variables are available for use in profile-cond:
```
input-type ( = "youtube"|"csv"|"string"|"bandcamp"|"spotify")
download-mode ( = "normal"|"aggregate"|"album"|"album-aggregate")
interactive (bool)
```
## Examples
Download tracks from a csv file:
```
sldl test.csv
```
<details>
<summary>CSV details</summary>
The names of the columns in the csv should be: `Artist`, `Title`, `Album`, `Length`. Some alternatives are also accepted. You can use `--print tracks` before downloading to check if everything has been parsed correctly. Only the title or album column is required, but additional info may improve search results.
</details>
<br>
Download spotify likes while skipping songs that already exist in the output folder:
```
sldl spotify-likes --skip-existing
```
<details>
<summary>Spotify details</summary>
To download private playlists or liked songs you will need to provide a client id and secret, which you can get here https://developer.spotify.com/dashboard/applications. Create an app and add `http://localhost:48721/callback` as a redirect url in its settings.
</details>
<br>
Download from a youtube playlist with fallback to yt-dlp in case it is not found on soulseek, and retrieve deleted video titles from wayback machine:
```
sldl "https://www.youtube.com/playlist?list=PLI_eFW8NAFzYAXZ5DrU6E6mQ_XfhaLBUX" --get-deleted --yt-dlp
```
<details>
<summary>YouTube details</summary>
Playlists are retrieved using the YoutubeExplode library which unfortunately doesn't always return all videos. You can use the official API by providing a key with `--youtube-key`. Get it here https://console.cloud.google.com. Create a new project, click "Enable Api" and search for "youtube data", then follow the prompts.
Also note that due the high number of music videos in the above example playlist, it may be better to remove all text in parentheses and disable song duration checking: `--regex "[\[\(].*?[\]\)]" --pref-length-tol -1`.
</details>
<br>
Search & download a specific song, preferring lossless:
```
sldl "title=MC MENTAL @ HIS BEST,length=242" --pref-format "flac,wav"
```
<details>
<summary>String details</summary>
The shorthand `sldl "Artist - Title"` is equivalent to `sldl "artist=Artist,title=Title"`.
</details>
<br>
Interactive album download:
```
sldl "album=Some Album" --interactive
```
<details>
<summary>Album details</summary>
The shorthand `sldl "Artist - Album" -a` is equivalent to `sldl "artist=Artist,album=Album"`. It's often helpful to restrict to folders which have two or more tracks: `--atc 2+`.
</details>
<br>
Print all songs by an artist which are not in your library:
```
sldl "artist=MC MENTAL" --aggregate --skip-existing --music-dir "path/to/music" --print tracks-full
```
<br>
## Download Modes
Depending on the provided input, the download behaviour changes:
- Normal download: The program will download a single file for every input entry.
- Album download: The program will search for the album and download an entire folder including non-audio files. Activated when the input is a link to a spotify or bandcamp album, when the input string or csv row has no track title, or when `-a`/`--album` is enabled.
- Aggregate download: With `-g`/`--aggregate`, the program will first perform an ordinary search for the input, then attempt to group the results into distinct songs and download one of each kind. This can be used to download an artist's entire discography (or simply printing it, like in the example above), finding remixes of a song, etc. Note that it is not 100% reliable, which is why `--min-users-aggregate` is set to 2 by default, i.e. any song that is shared by only one person will be ignored. Enable `--relax-filtering` to make the file filtering less aggressive.
## Options
Acronyms of two- and --three-word-arguments are also accepted, e.g. --twa
Download all albums:
```
Usage: sldl <input> [OPTIONS]
<input> <input> is one of the following:
Spotify playlist url or 'spotify-likes': Download a spotify
playlist or your liked songs. --spotify-id and
--spotify-secret may be required in addition.
Youtube playlist url: Download songs from a youtube playlist.
Provide a --youtube-key to include unavailabe uploads.
Path to a local CSV file: Use a csv file containing track
info to download. The names of the columns should be Artist,
Title, Album, Length. Only the title or album column is
required, but extra info may improve search results.
Name of the track, album, or artist to search for:
Can either be any typical search string or a comma-separated
list like 'title=Song Name,artist=Artist Name,length=215'
Allowed properties are: title, artist, album, length (sec)
Specify artist and album only to download an album.
Options:
--user <username> Soulseek username
--pass <password> Soulseek password
-p --path <path> Download directory
-f --folder <name> Subfolder name. Set to '.' to output directly to the
download folder (default: playlist/csv name)
-n --number <maxtracks> Download the first n tracks of a playlist
-o --offset <offset> Skip a specified number of tracks
-r --reverse Download tracks in reverse order
--name-format <format> Name format for downloaded tracks, e.g "{artist} - {title}"
--fast-search Begin downloading as soon as a file satisfying the preferred
conditions is found. Higher chance to download wrong files.
--remove-from-source Remove downloaded tracks from source playlist or CSV file
(spotify and CSV only)
--m3u <option> Create an m3u8 playlist file
'none': Do not create a playlist file
'fails' (default): Write only failed downloads to the m3u
'all': Write successes + fails as comments
--spotify-id <id> spotify client ID
--spotify-secret <secret> spotify client secret
--youtube-key <key> Youtube data API key
--get-deleted Attempt to retrieve titles of deleted videos from wayback
machine. Requires yt-dlp.
--deleted-only Only retrieve & download deleted music. Combine with --print
tracks-full to display a list of all deleted titles & urls.
--time-format <format> Time format in Length column of the csv file (e.g h:m:s.ms
for durations like 1:04:35.123). Default: s
--yt-parse Enable if the csv file contains YouTube video titles and
channel names; attempt to parse them into title and artist
names.
--format <format> Accepted file format(s), comma-separated
--length-tol <sec> Length tolerance in seconds
--min-bitrate <rate> Minimum file bitrate
--max-bitrate <rate> Maximum file bitrate
--min-samplerate <rate> Minimum file sample rate
--max-samplerate <rate> Maximum file sample rate
--min-bitdepth <depth> Minimum bit depth
--max-bitdepth <depth> Maximum bit depth
--banned-users <list> Comma-separated list of users to ignore
--pref-format <format> Preferred file format(s), comma-separated (default: mp3)
--pref-length-tol <sec> Preferred length tolerance in seconds (default: 3)
--pref-min-bitrate <rate> Preferred minimum bitrate (default: 200)
--pref-max-bitrate <rate> Preferred maximum bitrate (default: 2500)
--pref-min-samplerate <rate> Preferred minimum sample rate
--pref-max-samplerate <rate> Preferred maximum sample rate (default: 48000)
--pref-min-bitdepth <depth> Preferred minimum bit depth
--pref-max-bitdepth <depth> Preferred maximum bit depth
--pref-banned-users <list> Comma-separated list of users to deprioritize
--strict-conditions Skip files with missing properties instead of accepting by
default; if --min-bitrate is set, ignores any files with
unknown bitrate.
-a --album Album download mode
-t --interactive When downloading albums: Allows to select the wanted album
--album-track-count <num> Specify the exact number of tracks in the album. Folders
with a different number of tracks will be ignored. Append
a '+' or '-' after the number for the inequalities >= and <=
--album-ignore-fails When downloading an album and one of the files fails, do not
skip to the next source and do not delete all successfully
downloaded files
--album-art <option> Retrieve additional images after downloading the album:
'largest': Download from the folder with the largest image
'most': Download from the folder containing the most images
--album-art-only Only download album art for the provided album
-g --aggregate Instead of downloading a single track matching the input,
find and download all distinct songs associated with the
provided artist, album, or track title.
--min-users-aggregate <num> Minimum number of users sharing a track before it is
downloaded in aggregate mode. Setting it to higher values
will significantly reduce false positives, but also cause it
to ignore rarer songs. Default: 2
--relax-filtering Slightly relax file filtering in aggregate mode to include
more results
-s --skip-existing Skip if a track matching file conditions is found in the
output folder or your music library (if provided)
--skip-mode <mode> 'name': Use only filenames to check if a track exists
'name-precise' (default): Use filenames and check conditions
'tag': Use file tags (slower)
'tag-precise': Use file tags and check file conditions
'm3u': Skip all tracks that don't have a fail entry in m3u
--music-dir <path> Specify to also skip downloading tracks found in a music
library. Use with --skip-existing
--skip-not-found Skip searching for tracks that weren't found on Soulseek
during the last run. Fails are read from the m3u file.
--no-remove-special-chars Do not remove special characters before searching
--remove-ft Remove 'feat.' and everything after before searching
--remove-brackets Remove square brackets and their contents before searching
--regex <regex> Remove a regexp from all track titles and artist names.
Optionally specify a replacement regex after a semicolon.
Add 'T:', 'A:' or 'L:' at the start to only apply this to
the track title, artist, or album respectively.
--artist-maybe-wrong Performs an additional search without the artist name.
Useful for sources like SoundCloud where the "artist"
could just be an uploader. Note that when downloading a
YouTube playlist via url, this option is set automatically
on a per-track basis, so it is best kept off in that case.
-d --desperate Tries harder to find the desired track by searching for the
artist/album/title only, then filtering. (slower search)
--yt-dlp Use yt-dlp to download tracks that weren't found on
Soulseek. yt-dlp must be available from the command line.
--yt-dlp-argument <str> The command line arguments when running yt-dlp. Default:
"{id}" -f bestaudio/best -cix -o "{savepath}.%(ext)s"
Available vars are: {id}, {savedir}, {savepath} (w/o ext).
Note that with -x, yt-dlp will download webms in case
ffmpeg is unavailable.
-c --config <path> Set config file location
--search-timeout <ms> Max search time in ms (default: 5000)
--max-stale-time <ms> Max download time without progress in ms (default: 50000)
--concurrent-downloads <num> Max concurrent downloads (default: 2)
--searches-per-time <num> Max searches per time interval. Higher values may cause
30-minute bans. (default: 34)
--searches-renew-time <sec> Controls how often available searches are replenished.
Lower values may cause 30-minute bans. (default: 220)
--display-mode <option> Changes how searches and downloads are displayed:
'single' (default): Show transfer state and percentage
'double': Transfer state and a large progress bar
'simple': No download bars or changing percentages
--listen-port <port> Port for incoming connections (default: 50000)
--print <option> Print tracks or search results instead of downloading:
'tracks': Print all tracks to be downloaded
'tracks-full': Print extended information about all tracks
'results': Print search results satisfying file conditions
'results-full': Print search results including full paths
--debug Print extra debug info
```
### File conditions
Files not satisfying the conditions will not be downloaded. Files satisfying `pref-` conditions will be preferred; setting `--pref-format "flac,wav"` will make it download high quality files if they exist, and only download low quality files if there's nothing else. Conditions can also be supplied as a semicolon-delimited string to `--cond` and `--pref`, e.g `--cond "br>=320;f=mp3,ogg;sr<96000"`. See the start of `Program.cs` for the default file conditions.
**Important note**: Some info may be unavailable depending on the client used by the peer. For example, the default Soulseek client does not share the file bitrate. By default, if `--min-bitrate` is set, then files with unknown bitrate will still be downloaded. You can configure it to reject all files where one of the checked properties is unavailable by enabling `--strict-conditions`. (As a consequence, if `--min-bitrate` is also set then any files shared by users with the default client will be ignored)
### Name format
Variables enclosed in {} will be replaced by the corresponding file tag value. Available variables are: artist, sartist, artists, albumartist, albumartists, title, stitle, album, salbum, year, track, disc, filename, foldername. The variables sartist, stitle and salbum will be replaced by the source artist, title and album respectively (i.e what is shown in spotify/youtube/csv file) instead of the tag values of the downloaded file. Name format supports subdirectories as well as conditional expressions like `{tag1|tag2}` – If tag1 is null, choose tag2. String literals enclosed in parentheses are ignored in the null check. Examples:
- `{artist} - {title}`: Always name it 'Artist - Title'. Because some files on Soulseek do not have tags, the second example is preferred:
- `{artist( - )title|filename}`: If artist and title is not null, name it 'Artist - Title', otherwise use the original filename.
### Quality vs Speed
The following options will make it go faster, but may decrease search result quality or cause instability:
- `--fast-search` skips waiting until the search completes and downloads as soon as a file matching the preferred conditions is found
- `--concurrent-downloads` - set it to 4 or more
- `--max-stale-time` is set to 50 seconds by default, so it will wait a long time before giving up on a file
- `--searches-per-time` increase at the risk of ban, see the notes section for details.
### Quality vs Quantity
The options `--strict-title`, `--strict-artist` and `--strict-album` will filter any file that does not contain the title/artist/album in the filename (ignoring case, bounded by boundary chars). Another way to prevent false downloads is to set `--length-tol` to 3 or less to make it ignore any songs that differ from the input by more than 3 seconds. However, all 4 options are already enabled as 'preferred' conditions by default, meaning that such files will only be downloaded as a last resort even without enabling these. Hence it is only recommended to enable them if you need to minimize false downloads as much as possible.
## Configuration
Create a file named `sldl.conf` in the same directory as the executable and write your arguments there, e.g:
```
username = fakename
password = fakepass
pref-format = flac
fast-search = true
sldl "artist=MC MENTAL" --aggregate --album
```
## Notes
- For macOS builds you can use publish.sh to build the app. Download dotnet from https://dotnet.microsoft.com/en-us/download/dotnet/6.0, then run `chmod +x publish.sh && sh publish.sh`. For intel macs, uncomment the x64 and comment the arm64 section in publish.sh.
- `--display single` and especially `double` can cause the printed lines to be duplicated or overwritten on some configurations. Use `simple` if that's an issue.
- The server will ban you for 30 minutes if too many searches are performed within a short timespan. The program has a search limiter which can be adjusted with `--searches-per-time` and `--searches-renew-time` (when limit is reached, the status of the downloads will be "Waiting"). By default it is configured to allow up to 34 searches every 220 seconds. These values were determined through experimentation as unfortunately I couldn't find any information regarding soulseek's rate limits, so they may be incorrect.

2
publish.bat
View file

@ -1,6 +1,8 @@
@echo off
setlocal
set DOTNET_CLI_TELEMETRY_OPTOUT=1
if not exist slsk-batchdl\bin\zips mkdir slsk-batchdl\bin\zips
REM win-x86

1167
slsk-batchdl/Config.cs Normal file
View file

File diff suppressed because it is too large Load diff

324
slsk-batchdl/Data.cs Normal file
View file

@ -0,0 +1,324 @@
using Enums;
using SlDictionary = System.Collections.Concurrent.ConcurrentDictionary<string, (Soulseek.SearchResponse, Soulseek.File)>;
namespace Data
{
public class Track
{
public string Title = "";
public string Artist = "";
public string Album = "";
public string URI = "";
public int Length = -1;
public bool ArtistMaybeWrong = false;
public bool IsAlbum = false;
public int MinAlbumTrackCount = -1;
public int MaxAlbumTrackCount = -1;
public bool IsNotAudio = false;
public string DownloadPath = "";
public string Other = "";
public int CsvRow = -1;
public FailureReason FailureReason = FailureReason.None;
public TrackState State = TrackState.Initial;
public SlDictionary? Downloads = null;
public Track() { }
public Track(Track other)
{
Title = other.Title;
Artist = other.Artist;
Album = other.Album;
Length = other.Length;
URI = other.URI;
ArtistMaybeWrong = other.ArtistMaybeWrong;
Downloads = other.Downloads;
IsAlbum = other.IsAlbum;
IsNotAudio = other.IsNotAudio;
State = other.State;
FailureReason = other.FailureReason;
DownloadPath = other.DownloadPath;
Other = other.Other;
MinAlbumTrackCount = other.MinAlbumTrackCount;
MaxAlbumTrackCount = other.MaxAlbumTrackCount;
CsvRow = other.CsvRow;
}
public string ToKey()
{
return $"{Artist};{Album};{Title};{Length}";
}
public override string ToString()
{
return ToString(false);
}
public string ToString(bool noInfo = false)
{
if (IsNotAudio && Downloads != null && !Downloads.IsEmpty)
return $"{Utils.GetFileNameSlsk(Downloads.First().Value.Item2.Filename)}";
string str = Artist;
if (!IsAlbum && Title.Length == 0 && Downloads != null && !Downloads.IsEmpty)
{
str = $"{Utils.GetFileNameSlsk(Downloads.First().Value.Item2.Filename)}";
}
else if (Title.Length > 0 || Album.Length > 0)
{
if (str.Length > 0)
str += " - ";
if (IsAlbum)
str += Album;
else if (Title.Length > 0)
str += Title;
if (!noInfo)
{
if (Length > 0)
str += $" ({Length}s)";
if (IsAlbum)
str += " (album)";
}
}
else if (!noInfo)
{
str += " (artist)";
}
return str;
}
}
public class TrackListEntry
{
public List<List<Track>> list;
public ListType type;
public Track source;
public bool needSearch;
public bool placeInSubdir;
public TrackListEntry(List<List<Track>> list, ListType type, Track source)
{
this.list = list;
this.type = type;
this.source = source;
needSearch = type != ListType.Normal;
placeInSubdir = false;
}
public TrackListEntry(List<List<Track>> list, ListType type, Track source, bool needSearch, bool placeInSubdir)
{
this.list = list;
this.type = type;
this.source = source;
this.needSearch = needSearch;
this.placeInSubdir = placeInSubdir;
}
}
public class TrackLists
{
public List<TrackListEntry> lists = new();
public TrackLists() { }
public TrackLists(List<(List<List<Track>> list, ListType type, Track source)> lists)
{
foreach (var (list, type, source) in lists)
{
var newList = new List<List<Track>>();
foreach (var innerList in list)
{
var innerNewList = new List<Track>(innerList);
newList.Add(innerNewList);
}
this.lists.Add(new TrackListEntry(newList, type, source));
}
}
public static TrackLists FromFlattened(IEnumerable<Track> flatList, bool aggregate, bool album)
{
var res = new TrackLists();
using var enumerator = flatList.GetEnumerator();
while (enumerator.MoveNext())
{
var track = enumerator.Current;
if (album && aggregate)
{
res.AddEntry(ListType.AlbumAggregate, track);
}
else if (aggregate)
{
res.AddEntry(ListType.Aggregate, track);
}
else if (album || track.IsAlbum)
{
track.IsAlbum = true;
res.AddEntry(ListType.Album, track);
}
else
{
res.AddEntry(ListType.Normal);
res.AddTrackToLast(track);
bool hasNext;
while (true)
{
hasNext = enumerator.MoveNext();
if (!hasNext || enumerator.Current.IsAlbum)
break;
res.AddTrackToLast(enumerator.Current);
}
if (hasNext && enumerator.Current.IsAlbum)
res.AddEntry(ListType.Album, track);
else if (!hasNext)
break;
}
}
return res;
}
public TrackListEntry this[int index]
{
get { return lists[index]; }
set { lists[index] = value; }
}
public void AddEntry(TrackListEntry tle)
{
lists.Add(tle);
}
public void AddEntry(List<List<Track>>? list, ListType? type = null, Track? source = null)
{
type ??= ListType.Normal;
source ??= new Track();
list ??= new List<List<Track>>();
lists.Add(new TrackListEntry(list, (ListType)type, source));
}
public void AddEntry(List<Track> tracks, ListType? type = null, Track? source = null)
{
var list = new List<List<Track>>() { tracks };
AddEntry(list, type, source);
}
public void AddEntry(Track track, ListType? type = null, Track? source = null)
{
var list = new List<List<Track>>() { new List<Track>() { track } };
AddEntry(list, type, source);
}
public void AddEntry(ListType? type = null, Track? source = null)
{
var list = new List<List<Track>>() { new List<Track>() };
AddEntry(list, type, source);
}
public void AddTrackToLast(Track track)
{
int i = lists.Count - 1;
int j = lists[i].list.Count - 1;
lists[i].list[j].Add(track);
}
public void Reverse()
{
lists.Reverse();
foreach (var tle in lists)
{
foreach (var ls in tle.list)
{
ls.Reverse();
}
}
}
public IEnumerable<Track> Flattened(bool addSources, bool addSpecialSourceTracks, bool sourcesOnly = false)
{
foreach (var tle in lists)
{
if ((addSources || sourcesOnly) && tle.source != null)
yield return tle.source;
if (!sourcesOnly && tle.list.Count > 0 && (tle.type == ListType.Normal || addSpecialSourceTracks))
{
foreach (var t in tle.list[0])
yield return t;
}
}
}
}
public class TrackStringComparer : IEqualityComparer<Track>
{
private bool _ignoreCase = false;
public TrackStringComparer(bool ignoreCase = false)
{
_ignoreCase = ignoreCase;
}
public bool Equals(Track a, Track b)
{
if (a.Equals(b))
return true;
var comparer = _ignoreCase ? StringComparison.OrdinalIgnoreCase : StringComparison.Ordinal;
return string.Equals(a.Title, b.Title, comparer)
&& string.Equals(a.Artist, b.Artist, comparer)
&& string.Equals(a.Album, b.Album, comparer);
}
public int GetHashCode(Track a)
{
unchecked
{
int hash = 17;
string trackTitle = _ignoreCase ? a.Title.ToLower() : a.Title;
string artistName = _ignoreCase ? a.Artist.ToLower() : a.Artist;
string album = _ignoreCase ? a.Album.ToLower() : a.Album;
hash = hash * 23 + trackTitle.GetHashCode();
hash = hash * 23 + artistName.GetHashCode();
hash = hash * 23 + album.GetHashCode();
return hash;
}
}
}
public class SimpleFile
{
public string Path;
public string? Artists;
public string? Title;
public string? Album;
public int Length;
public int Bitrate;
public int Samplerate;
public int Bitdepth;
public SimpleFile(TagLib.File file)
{
Path = file.Name;
Artists = file.Tag.JoinedPerformers;
Title = file.Tag.Title;
Album = file.Tag.Album;
Length = (int)file.Length;
Bitrate = file.Properties.AudioBitrate;
Samplerate = file.Properties.AudioSampleRate;
Bitdepth = file.Properties.BitsPerSample;
}
}
public class ResponseData
{
public int lockedFilesCount;
}
}

89
slsk-batchdl/Enums.cs Normal file
View file

@ -0,0 +1,89 @@
namespace Enums
{
public enum FailureReason
{
None = 0,
InvalidSearchString = 1,
OutOfDownloadRetries = 2,
NoSuitableFileFound = 3,
AllDownloadsFailed = 4,
}
public enum TrackState
{
Initial = 0,
Downloaded = 1,
Failed = 2,
AlreadyExists = 3,
NotFoundLastTime = 4
}
public enum SkipMode
{
Name = 0,
NameCond = 1,
Tag = 2,
TagCond = 3,
// non file-based skip modes are >= 4
M3u = 4,
M3uCond = 5,
}
public enum InputType
{
CSV,
YouTube,
Spotify,
Bandcamp,
String,
None,
}
public enum ListType
{
Normal,
Album,
Aggregate,
AlbumAggregate,
}
public enum M3uOption
{
None,
Index,
All,
}
public enum PrintOption
{
None = 0,
Tracks = 1,
Results = 2,
Full = 4,
}
public enum AlbumArtOption
{
Default,
Most,
Largest,
MostLargest,
}
public enum DisplayMode
{
Single,
Double,
Simple,
}
public enum Verbosity
{
Silent,
Error,
Warning,
Normal,
Verbose
}
}

351
slsk-batchdl/ExistingChecker.cs Normal file
View file

@ -0,0 +1,351 @@

using Data;
using Enums;
using System.IO;
namespace ExistingCheckers
{
public static class Registry
{
static IExistingChecker GetChecker(SkipMode mode, string dir, FileConditions conditions, M3uEditor m3uEditor)
{
bool noConditions = conditions.Equals(new FileConditions());
return mode switch
{
SkipMode.Name => new NameExistingChecker(dir),
SkipMode.NameCond => noConditions ? new NameExistingChecker(dir) : new NameConditionExistingChecker(dir, conditions),
SkipMode.Tag => new TagExistingChecker(dir),
SkipMode.TagCond => noConditions ? new TagExistingChecker(dir) : new TagConditionExistingChecker(dir, conditions),
SkipMode.M3u => new M3uExistingChecker(m3uEditor, false),
SkipMode.M3uCond => noConditions ? new M3uExistingChecker(m3uEditor, true) : new M3uConditionExistingChecker(m3uEditor, conditions),
};
}
public static Dictionary<Track, string> SkipExisting(List<Track> tracks, string dir, FileConditions necessaryCond, M3uEditor m3uEditor, SkipMode mode)
{
var existing = new Dictionary<Track, string>();
var checker = GetChecker(mode, dir, necessaryCond, m3uEditor);
checker.BuildIndex();
for (int i = 0; i < tracks.Count; i++)
{
if (tracks[i].IsNotAudio)
continue;
if (checker.TrackExists(tracks[i], out string? path))
{
existing.TryAdd(tracks[i], path);
tracks[i].State = TrackState.AlreadyExists;
tracks[i].DownloadPath = path;
}
}
return existing;
}
}
public interface IExistingChecker
{
public bool TrackExists(Track track, out string? foundPath);
public void BuildIndex() { }
}
public class NameExistingChecker : IExistingChecker
{
readonly string[] ignore = new string[] { " ", "_", "-", ".", "(", ")", "[", "]" };
readonly string dir;
readonly List<(string, string, string)> index = new(); // (Path, PreprocessedPath, PreprocessedName)
public NameExistingChecker(string dir)
{
this.dir = dir;
}
private string Preprocess(string s, bool removeSlash)
{
s = s.ToLower().Replace(ignore, "");
s = s.ReplaceInvalidChars("", false, removeSlash);
s = s.RemoveFt();
s = s.RemoveDiacritics();
return s;
}
public void BuildIndex()
{
var files = Directory.GetFiles(dir, "*", SearchOption.AllDirectories);
int removeLen = Preprocess(dir, false).Length + 1;
foreach (var path in files)
{
if (Utils.IsMusicFile(path))
{
string ppath = Preprocess(path[..path.LastIndexOf('.')], false)[removeLen..];
string pname = Path.GetFileName(ppath);
index.Add((path, ppath, pname));
}
}
}
public bool TrackExists(Track track, out string? foundPath)
{
string title = Preprocess(track.Title, true);
string artist = Preprocess(track.Artist, true);
foreach ((var path, var ppath, var pname) in index)
{
if (pname.Contains(title) && ppath.Contains(artist))
{
foundPath = path;
return true;
}
}
foundPath = null;
return false;
}
}
public class NameConditionExistingChecker : IExistingChecker
{
readonly string[] ignore = new string[] { " ", "_", "-", ".", "(", ")", "[", "]" };
readonly string dir;
readonly List<(string, string, SimpleFile)> index = new(); // (PreprocessedPath, PreprocessedName, file)
FileConditions conditions;
public NameConditionExistingChecker(string dir, FileConditions conditions)
{
this.dir = dir;
this.conditions = conditions;
}
private string Preprocess(string s, bool removeSlash)
{
s = s.ToLower().Replace(ignore, "");
s = s.ReplaceInvalidChars("", false, removeSlash);
s = s.RemoveFt();
s = s.RemoveDiacritics();
return s;
}
public void BuildIndex()
{
var files = Directory.GetFiles(dir, "*", SearchOption.AllDirectories);
int removeLen = Preprocess(dir, false).Length + 1;
foreach (var path in files)
{
if (Utils.IsMusicFile(path))
{
TagLib.File musicFile;
try { musicFile = TagLib.File.Create(path); }
catch { continue; }
string ppath = Preprocess(path[..path.LastIndexOf('.')], false)[removeLen..];
string pname = Path.GetFileName(ppath);
index.Add((ppath, pname, new SimpleFile(musicFile)));
}
}
}
public bool TrackExists(Track track, out string? foundPath)
{
string title = Preprocess(track.Title, true);
string artist = Preprocess(track.Artist, true);
foreach ((var ppath, var pname, var musicFile) in index)
{
if (pname.Contains(title) && ppath.Contains(artist) && conditions.FileSatisfies(musicFile, track))
{
foundPath = musicFile.Path;
return true;
}
}
foundPath = null;
return false;
}
}
public class TagExistingChecker : IExistingChecker
{
readonly string dir;
readonly List<(string, string, string)> index = new(); // (Path, PreprocessedArtist, PreprocessedTitle)
public TagExistingChecker(string dir)
{
this.dir = dir;
}
private string Preprocess(string s)
{
return s.Replace(" ", "").RemoveFt().ToLower();
}
public void BuildIndex()
{
var files = Directory.GetFiles(dir, "*", SearchOption.AllDirectories);
foreach (var path in files)
{
if (Utils.IsMusicFile(path))
{
TagLib.File musicFile;
try { musicFile = TagLib.File.Create(path); }
catch { continue; }
string partist = Preprocess(musicFile.Tag.JoinedPerformers ?? "");
string ptitle = Preprocess(musicFile.Tag.Title ?? "");
index.Add((path, partist, ptitle));
}
}
}
public bool TrackExists(Track track, out string? foundPath)
{
string title = Preprocess(track.Title);
string artist = Preprocess(track.Artist);
foreach ((var path, var partist, var ptitle) in index)
{
if (title==ptitle && partist.Contains(artist))
{
foundPath = path;
return true;
}
}
foundPath = null;
return false;
}
}
public class TagConditionExistingChecker : IExistingChecker
{
readonly string dir;
readonly List<(string, string, SimpleFile)> index = new(); // (PreprocessedArtist, PreprocessedTitle, file)
FileConditions conditions;
public TagConditionExistingChecker(string dir, FileConditions conditions)
{
this.dir = dir;
this.conditions = conditions;
}
private string Preprocess(string s)
{
return s.Replace(" ", "").RemoveFt().ToLower();
}
public void BuildIndex()
{
var files = Directory.GetFiles(dir, "*", SearchOption.AllDirectories);
foreach (var path in files)
{
if (Utils.IsMusicFile(path))
{
TagLib.File musicFile;
try { musicFile = TagLib.File.Create(path); }
catch { continue; }
string partist = Preprocess(musicFile.Tag.JoinedPerformers ?? "");
string ptitle = Preprocess(musicFile.Tag.Title ?? "");
index.Add((partist, ptitle, new SimpleFile(musicFile)));
}
}
}
public bool TrackExists(Track track, out string? foundPath)
{
string title = Preprocess(track.Title);
string artist = Preprocess(track.Artist);
foreach ((var partist, var ptitle, var musicFile) in index)
{
if (title == ptitle && partist.Contains(artist) && conditions.FileSatisfies(musicFile, track))
{
foundPath = musicFile.Path;
return true;
}
}
foundPath = null;
return false;
}
}
public class M3uExistingChecker : IExistingChecker
{
M3uEditor m3uEditor;
bool checkFileExists;
public M3uExistingChecker(M3uEditor m3UEditor, bool checkFileExists)
{
this.m3uEditor = m3UEditor;
this.checkFileExists = checkFileExists;
}
public bool TrackExists(Track track, out string? foundPath)
{
foundPath = null;
var t = m3uEditor.PreviousRunResult(track);
if (t != null && (t.State == TrackState.Downloaded || t.State == TrackState.AlreadyExists))
{
if (checkFileExists && (t.DownloadPath.Length == 0 || !File.Exists(t.DownloadPath)))
{
return false;
}
foundPath = t.DownloadPath;
return true;
}
return false;
}
}
public class M3uConditionExistingChecker : IExistingChecker
{
M3uEditor m3uEditor;
FileConditions conditions;
public M3uConditionExistingChecker(M3uEditor m3UEditor, FileConditions conditions)
{
this.m3uEditor = m3UEditor;
this.conditions = conditions;
}
public bool TrackExists(Track track, out string? foundPath)
{
foundPath = null;
var t = m3uEditor.PreviousRunResult(track);
if (t != null && (t.State == TrackState.Downloaded || t.State == TrackState.AlreadyExists) && t.DownloadPath.Length > 0)
{
if (File.Exists(t.DownloadPath))
{
TagLib.File musicFile;
try
{
musicFile = TagLib.File.Create(t.DownloadPath);
if (conditions.FileSatisfies(musicFile, track, false))
{
foundPath = t.DownloadPath;
return true;
}
else
{
return false;
}
}
catch
{
return false;
}
}
}
return false;
}
}
}

109
slsk-batchdl/Extractors/Bandcamp.cs Normal file
View file

@ -0,0 +1,109 @@
using Data;
using HtmlAgilityPack;
using Enums;
using System.Net;
using System.Text.RegularExpressions;
using System.Text.Json;
namespace Extractors
{
public class BandcampExtractor : IExtractor
{
public static bool InputMatches(string input)
{
input = input.ToLower();
return input.IsInternetUrl() && input.Contains("bandcamp.com");
}
public async Task<TrackLists> GetTracks()
{
var trackLists = new TrackLists();
bool isTrack = Config.input.Contains("/track/");
bool isAlbum = !isTrack && Config.input.Contains("/album/");
bool isArtist =!isTrack && !isAlbum;
if (isArtist)
{
string artistUrl = Config.input.TrimEnd('/');
if (!artistUrl.EndsWith("/music"))
artistUrl += "/music";
using var httpClient = new HttpClient();
var response = await httpClient.GetStringAsync(artistUrl);
string idPattern = @"band_id=(\d+)&";
var match = Regex.Match(response, idPattern);
var id = match.Groups[1].Value;
var address = $"http://bandcamp.com/api/mobile/24/band_details?band_id={id}";
var responseString = await httpClient.GetStringAsync(address);
var jsonDocument = JsonDocument.Parse(responseString);
var root = jsonDocument.RootElement;
string artistName = root.GetProperty("name").GetString();
var tralbums = new List<Track>();
foreach (var item in root.GetProperty("discography").EnumerateArray())
{
//ItemType = item.GetProperty("item_type").GetString(),
var t = new Track()
{
Album = item.GetProperty("title").GetString(),
Artist = item.GetProperty("artist_name").GetString() ?? item.GetProperty("band_name").GetString(),
IsAlbum = true,
};
trackLists.AddEntry(ListType.Album, t);
}
}
else
{
var web = new HtmlWeb();
var doc = await web.LoadFromWebAsync(Config.input);
var nameSection = doc.DocumentNode.SelectSingleNode("//div[@id='name-section']");
var name = nameSection.SelectSingleNode(".//h2[contains(@class, 'trackTitle')]").InnerText.UnHtmlString().Trim();
if (isAlbum)
{
var artist = nameSection.SelectSingleNode(".//h3/span/a").InnerText.UnHtmlString().Trim();
var track = new Track() { Artist = artist, Album = name, IsAlbum = true };
trackLists.AddEntry(ListType.Album, track);
if (Config.setAlbumMinTrackCount || Config.setAlbumMaxTrackCount)
{
var trackTable = doc.DocumentNode.SelectSingleNode("//*[@id='track_table']");
int n = trackTable.SelectNodes(".//tr").Count;
if (Config.setAlbumMinTrackCount)
track.MinAlbumTrackCount = n;
if (Config.setAlbumMaxTrackCount)
track.MaxAlbumTrackCount = n;
}
Config.defaultFolderName = track.ToString(true).ReplaceInvalidChars(Config.invalidReplaceStr).Trim();
}
else
{
var album = nameSection.SelectSingleNode(".//h3[contains(@class, 'albumTitle')]/span/a").InnerText.UnHtmlString().Trim();
var artist = nameSection.SelectSingleNode(".//h3[contains(@class, 'albumTitle')]/span[last()]/a").InnerText.UnHtmlString().Trim();
//var timeParts = doc.DocumentNode.SelectSingleNode("//span[@class='time_total']").InnerText.Trim().Split(':');
var track = new Track() { Artist = artist, Title = name, Album = album };
trackLists.AddEntry(track);
Config.defaultFolderName = ".";
}
}
if (!Config.reverse)
{
trackLists = TrackLists.FromFlattened(trackLists.Flattened(true, false).Skip(Config.offset).Take(Config.maxTracks), Config.aggregate, Config.album);
}
return trackLists;
}
}
}

209
slsk-batchdl/Extractors/Csv.cs Normal file
View file

@ -0,0 +1,209 @@
using Data;
using System.Text.RegularExpressions;
namespace Extractors
{
public class CsvExtractor : IExtractor
{
object csvLock = new();
int csvColumnCount = -1;
public static bool InputMatches(string input)
{
input = input.ToLower();
return !input.IsInternetUrl() && input.EndsWith(".csv");
}
public async Task<TrackLists> GetTracks()
{
int max = Config.reverse ? int.MaxValue : Config.maxTracks;
int off = Config.reverse ? 0 : Config.offset;
if (!File.Exists(Config.input))
throw new FileNotFoundException("CSV file not found");
var tracks = await ParseCsvIntoTrackInfo(Config.input, Config.artistCol, Config.trackCol, Config.lengthCol, Config.albumCol, Config.descCol, Config.ytIdCol, Config.trackCountCol, Config.timeUnit, Config.ytParse);
var trackLists = TrackLists.FromFlattened(tracks.Skip(off).Take(max), Config.aggregate, Config.album);
Config.defaultFolderName = Path.GetFileNameWithoutExtension(Config.input);
return trackLists;
}
public async Task RemoveTrackFromSource(Track track)
{
lock (csvLock)
{
if (File.Exists(Config.input))
{
string[] lines = File.ReadAllLines(Config.input, System.Text.Encoding.UTF8);
if (lines.Length > track.CsvRow)
{
lines[track.CsvRow] = new string(',', Math.Max(0, csvColumnCount - 1));
Utils.WriteAllLines(Config.input, lines, '\n');
}
}
}
}
async Task<List<Track>> ParseCsvIntoTrackInfo(string path, string artistCol = "", string trackCol = "",
string lengthCol = "", string albumCol = "", string descCol = "", string ytIdCol = "", string trackCountCol = "", string timeUnit = "s", bool ytParse = false)
{
var tracks = new List<Track>();
using var sr = new StreamReader(path, System.Text.Encoding.UTF8);
var parser = new SmallestCSV.SmallestCSVParser(sr);
int index = 0;
var header = parser.ReadNextRow();
while (header == null || header.Count == 0 || !header.Any(t => t.Trim().Length > 0))
{
index++;
header = parser.ReadNextRow();
}
string[] cols = { artistCol, albumCol, trackCol, lengthCol, descCol, ytIdCol, trackCountCol };
string[][] aliases = {
new[] { "artist", "artist name", "artists", "artist names" },
new[] { "album", "album name", "album title" },
new[] { "title", "song", "track title", "track name", "song name", "track" },
new[] { "length", "duration", "track length", "track duration", "song length", "song duration" },
new[] { "description", "youtube description" },
new[] { "url", "id", "youtube id" },
new[] { "track count", "album track count" }
};
string usingColumns = "";
for (int i = 0; i < cols.Length; i++)
{
if (string.IsNullOrEmpty(cols[i]))
{
string? res = header.FirstOrDefault(h => Regex.Replace(h, @"\(.*?\)", "").Trim().EqualsAny(aliases[i], StringComparison.OrdinalIgnoreCase));
if (!string.IsNullOrEmpty(res))
{
cols[i] = res;
usingColumns += $"{aliases[i][0]}:\"{res}\", ";
}
}
else
{
if (header.IndexOf(cols[i]) == -1)
throw new Exception($"Column \"{cols[i]}\" not found in CSV file");
usingColumns += $"{aliases[i][0]}:\"{cols[i]}\", ";
}
}
int foundCount = cols.Count(col => col.Length > 0);
if (!string.IsNullOrEmpty(usingColumns))
Console.WriteLine($"Using columns: {usingColumns.TrimEnd(' ', ',')}.");
else if (foundCount == 0)
throw new Exception("No columns specified and couldn't determine automatically");
int[] indices = cols.Select(col => col.Length == 0 ? -1 : header.IndexOf(col)).ToArray();
int artistIndex, albumIndex, trackIndex, lengthIndex, descIndex, ytIdIndex, trackCountIndex;
(artistIndex, albumIndex, trackIndex, lengthIndex, descIndex, ytIdIndex, trackCountIndex) = (indices[0], indices[1], indices[2], indices[3], indices[4], indices[5], indices[6]);
while (true)
{
index++;
var values = parser.ReadNextRow();
if (values == null)
break;
if (!values.Any(t => t.Trim().Length > 0))
continue;
while (values.Count < foundCount)
values.Add("");
if (csvColumnCount == -1)
csvColumnCount = values.Count;
var desc = "";
var track = new Track() { CsvRow = index };
if (artistIndex >= 0) track.Artist = values[artistIndex];
if (trackIndex >= 0) track.Title = values[trackIndex];
if (albumIndex >= 0) track.Album = values[albumIndex];
if (descIndex >= 0) desc = values[descIndex];
if (ytIdIndex >= 0) track.URI = values[ytIdIndex];
if (trackCountIndex >= 0)
{
string a = values[trackCountIndex].Trim();
if (a == "-1")
{
track.MinAlbumTrackCount = -1;
track.MaxAlbumTrackCount = -1;
}
else if (a.Last() == '-' && int.TryParse(a.AsSpan(0, a.Length - 1), out int n))
{
track.MaxAlbumTrackCount = n;
}
else if (a.Last() == '+' && int.TryParse(a.AsSpan(0, a.Length - 1), out n))
{
track.MinAlbumTrackCount = n;
}
else if (int.TryParse(a, out n))
{
track.MinAlbumTrackCount = n;
track.MaxAlbumTrackCount = n;
}
}
if (lengthIndex >= 0)
{
try
{
track.Length = (int)ParseTrackLength(values[lengthIndex], timeUnit);
}
catch
{
Program.WriteLine($"Couldn't parse track length \"{values[lengthIndex]}\" with format \"{timeUnit}\" for \"{track}\"", ConsoleColor.DarkYellow);
}
}
if (ytParse)
track = await YouTube.ParseTrackInfo(track.Title, track.Artist, track.URI, track.Length, desc);
track.IsAlbum = track.Title.Length == 0 && track.Album.Length > 0;
if (track.Title.Length > 0 || track.Artist.Length > 0 || track.Album.Length > 0)
tracks.Add(track);
}
if (ytParse)
YouTube.StopService();
return tracks;
}
double ParseTrackLength(string duration, string format)
{
if (string.IsNullOrEmpty(format))
throw new ArgumentException("Duration format string empty");
duration = Regex.Replace(duration, "[a-zA-Z]", "");
var formatParts = Regex.Split(format, @"\W+");
var durationParts = Regex.Split(duration, @"\W+").Where(s => !string.IsNullOrEmpty(s)).ToArray();
double totalSeconds = 0;
for (int i = 0; i < formatParts.Length; i++)
{
switch (formatParts[i])
{
case "h":
totalSeconds += double.Parse(durationParts[i]) * 3600;
break;
case "m":
totalSeconds += double.Parse(durationParts[i]) * 60;
break;
case "s":
totalSeconds += double.Parse(durationParts[i]);
break;
case "ms":
totalSeconds += double.Parse(durationParts[i]) / Math.Pow(10, durationParts[i].Length);
break;
}
}
return totalSeconds;
}
}
}

341
slsk-batchdl/Extractors/Spotify.cs Normal file
View file

@ -0,0 +1,341 @@
using SpotifyAPI.Web;
using SpotifyAPI.Web.Auth;
using Swan;
using Data;
using Enums;
namespace Extractors
{
public class SpotifyExtractor : IExtractor
{
private Spotify? spotifyClient;
public string playlistUri = "";
public static bool InputMatches(string input)
{
input = input.ToLower();
return input == "spotify-likes" || input.IsInternetUrl() && input.Contains("spotify.com");
}
public async Task<TrackLists> GetTracks()
{
var trackLists = new TrackLists();
int max = Config.reverse ? int.MaxValue : Config.maxTracks;
int off = Config.reverse ? 0 : Config.offset;
string playlistName = "";
bool needLogin = Config.input == "spotify-likes" || Config.removeTracksFromSource;
List<Track> tracks = new List<Track>();
static void readSpotifyCreds()
{
Console.Write("Spotify client ID:");
Config.spotifyId = Console.ReadLine();
Console.Write("Spotify client secret:");
Config.spotifySecret = Console.ReadLine();
Console.WriteLine();
}
if (needLogin && (Config.spotifyId.Length == 0 || Config.spotifySecret.Length == 0))
{
readSpotifyCreds();
}
spotifyClient = new Spotify(Config.spotifyId, Config.spotifySecret);
await spotifyClient.Authorize(needLogin, Config.removeTracksFromSource);
if (Config.input == "spotify-likes")
{
Console.WriteLine("Loading Spotify likes");
tracks = await spotifyClient.GetLikes(max, off);
playlistName = "Spotify Likes";
trackLists.AddEntry(tracks);
if (Config.album || Config.aggregate)
trackLists = TrackLists.FromFlattened(trackLists.Flattened(true, false), Config.aggregate, Config.album);
}
else if (Config.input.Contains("/album/"))
{
Console.WriteLine("Loading Spotify album");
(var source, tracks) = await spotifyClient.GetAlbum(Config.input);
playlistName = source.ToString(noInfo: true);
trackLists.AddEntry(ListType.Album, source);
if (Config.setAlbumMinTrackCount)
source.MinAlbumTrackCount = tracks.Count;
if (Config.setAlbumMaxTrackCount)
source.MaxAlbumTrackCount = tracks.Count;
}
else if (Config.input.Contains("/artist/"))
{
Console.WriteLine("Loading spotify artist");
throw new NotImplementedException("Spotify artist download currently not supported.");
}
else
{
try
{
Console.WriteLine("Loading Spotify playlist");
(playlistName, playlistUri, tracks) = await spotifyClient.GetPlaylist(Config.input, max, off);
}
catch (SpotifyAPI.Web.APIException)
{
if (!needLogin && !spotifyClient.UsedDefaultCredentials)
{
await spotifyClient.Authorize(true, Config.removeTracksFromSource);
(playlistName, playlistUri, tracks) = await spotifyClient.GetPlaylist(Config.input, max, off);
}
else if (!needLogin)
{
Console.WriteLine("Spotify playlist not found. It may be set to private. Login? [Y/n]");
if (Console.ReadLine()?.ToLower().Trim() == "y")
{
readSpotifyCreds();
spotifyClient = new Spotify(Config.spotifyId, Config.spotifySecret);
await spotifyClient.Authorize(true, Config.removeTracksFromSource);
Console.WriteLine("Loading Spotify playlist");
(playlistName, playlistUri, tracks) = await spotifyClient.GetPlaylist(Config.input, max, off);
}
else
{
Environment.Exit(0);
}
}
else throw;
}
trackLists.AddEntry(tracks);
if (Config.album || Config.aggregate)
trackLists = TrackLists.FromFlattened(trackLists.Flattened(true, false), Config.aggregate, Config.album);
}
Config.defaultFolderName = playlistName.ReplaceInvalidChars(Config.invalidReplaceStr);
return trackLists;
}
public async Task RemoveTrackFromSource(Track track)
{
if (playlistUri.Length > 0 && track.URI.Length > 0)
await spotifyClient.RemoveTrackFromPlaylist(playlistUri, track.URI);
}
}
public class Spotify
{
private EmbedIOAuthServer _server;
private readonly string _clientId;
private readonly string _clientSecret;
private SpotifyClient? _client;
private bool loggedIn = false;
// default spotify credentials (base64-encoded to keep the bots away)
public const string encodedSpotifyId = "MWJmNDY5M1bLaH9WJiYjFhNGY0MWJjZWQ5YjJjMWNmZGJiZDI=";
public const string encodedSpotifySecret = "Y2JlM2QxYTE5MzJkNDQ2MmFiOGUy3shTuf4Y2JhY2M3ZDdjYWU=";
public bool UsedDefaultCredentials { get; private set; }
public Spotify(string clientId = "", string clientSecret = "")
{
_clientId = clientId;
_clientSecret = clientSecret;
if (_clientId.Length == 0 || _clientSecret.Length == 0)
{
_clientId = System.Text.Encoding.UTF8.GetString(Convert.FromBase64String(encodedSpotifyId.Replace("1bLaH9", "")));
_clientSecret = System.Text.Encoding.UTF8.GetString(Convert.FromBase64String(encodedSpotifySecret.Replace("3shTuf4", "")));
UsedDefaultCredentials = true;
}
}
public async Task Authorize(bool login = false, bool needModify = false)
{
_client = null;
if (!login)
{
var config = SpotifyClientConfig.CreateDefault();
var request = new ClientCredentialsRequest(_clientId, _clientSecret);
var response = await new OAuthClient(config).RequestToken(request);
_client = new SpotifyClient(config.WithToken(response.AccessToken));
}
else
{
Swan.Logging.Logger.NoLogging();
_server = new EmbedIOAuthServer(new Uri("http://localhost:48721/callback"), 48721);
await _server.Start();
_server.AuthorizationCodeReceived += OnAuthorizationCodeReceived;
_server.ErrorReceived += OnErrorReceived;
var scope = new List<string> {
Scopes.UserLibraryRead, Scopes.PlaylistReadPrivate, Scopes.PlaylistReadCollaborative
};
if (needModify)
{
scope.Add(Scopes.PlaylistModifyPublic);
scope.Add(Scopes.PlaylistModifyPrivate);
}
var request = new LoginRequest(_server.BaseUri, _clientId, LoginRequest.ResponseType.Code) { Scope = scope };
BrowserUtil.Open(request.ToUri());
await IsClientReady();
}
}
private async Task OnAuthorizationCodeReceived(object sender, AuthorizationCodeResponse response)
{
await _server.Stop();
var config = SpotifyClientConfig.CreateDefault();
var tokenResponse = await new OAuthClient(config).RequestToken(
new AuthorizationCodeTokenRequest(
_clientId, _clientSecret, response.Code, new Uri("http://localhost:48721/callback")
)
);
_client = new SpotifyClient(tokenResponse.AccessToken);
loggedIn = true;
}
private async Task OnErrorReceived(object sender, string error, string state)
{
await _server.Stop();
throw new Exception($"Aborting authorization, error received: {error}");
}
public async Task<bool> IsClientReady()
{
while (_client == null)
await Task.Delay(1000);
return true;
}
public async Task<List<Track>> GetLikes(int max = int.MaxValue, int offset = 0)
{
if (!loggedIn)
throw new Exception("Can't get liked music as user is not logged in");
List<Track> res = new List<Track>();
int limit = Math.Min(max, 50);
while (true)
{
var tracks = await _client.Library.GetTracks(new LibraryTracksRequest { Limit = limit, Offset = offset });
foreach (var track in tracks.Items)
{
string[] artists = ((IEnumerable<object>)track.Track.ReadProperty("artists")).Select(a => (string)a.ReadProperty("name")).ToArray();
string artist = artists[0];
string name = (string)track.Track.ReadProperty("name");
string album = (string)track.Track.ReadProperty("album").ReadProperty("name");
int duration = (int)track.Track.ReadProperty("durationMs");
res.Add(new Track { Album = album, Artist = artist, Title = name, Length = duration / 1000 });
}
if (tracks.Items.Count < limit || res.Count >= max)
break;
offset += limit;
limit = Math.Min(max - res.Count, 50);
}
return res;
}
public async Task RemoveTrackFromPlaylist(string playlistId, string trackUri)
{
var item = new PlaylistRemoveItemsRequest.Item { Uri = trackUri };
var pr = new PlaylistRemoveItemsRequest();
pr.Tracks = new List<PlaylistRemoveItemsRequest.Item>() { item };
try { await _client.Playlists.RemoveItems(playlistId, pr); }
catch { }
}
public async Task<(string?, string?, List<Track>)> GetPlaylist(string url, int max = int.MaxValue, int offset = 0)
{
var playlistId = GetPlaylistIdFromUrl(url);
var p = await _client.Playlists.Get(playlistId);
List<Track> res = new List<Track>();
int limit = Math.Min(max, 100);
while (true)
{
var tracks = await _client.Playlists.GetItems(playlistId, new PlaylistGetItemsRequest { Limit = limit, Offset = offset });
foreach (var track in tracks.Items)
{
try
{
string[] artists = ((IEnumerable<object>)track.Track.ReadProperty("artists")).Select(a => (string)a.ReadProperty("name")).ToArray();
var t = new Track()
{
Artist = artists[0],
Album = (string)track.Track.ReadProperty("album").ReadProperty("name"),
Title = (string)track.Track.ReadProperty("name"),
Length = (int)track.Track.ReadProperty("durationMs") / 1000,
URI = (string)track.Track.ReadProperty("uri"),
};
res.Add(t);
}
catch
{
continue;
}
}
if (tracks.Items.Count < limit || res.Count >= max)
break;
offset += limit;
limit = Math.Min(max - res.Count, 100);
}
return (p.Name, p.Id, res);
}
private string GetPlaylistIdFromUrl(string url)
{
var uri = new Uri(url);
var segments = uri.Segments;
return segments[segments.Length - 1].TrimEnd('/');
}
public async Task<(Track, List<Track>)> GetAlbum(string url)
{
var albumId = GetAlbumIdFromUrl(url);
var album = await _client.Albums.Get(albumId);
List<Track> tracks = new List<Track>();
foreach (var track in album.Tracks.Items)
{
var t = new Track()
{
Album = album.Name,
Artist = track.Artists.First().Name,
Title = track.Name,
Length = track.DurationMs / 1000,
URI = track.Uri,
};
tracks.Add(t);
}
return (new Track { Album = album.Name, Artist = album.Artists.First().Name, IsAlbum = true }, tracks);
}
private string GetAlbumIdFromUrl(string url)
{
var uri = new Uri(url);
var segments = uri.Segments;
return segments[^1].TrimEnd('/');
}
}
}

169
slsk-batchdl/Extractors/String.cs Normal file
View file

@ -0,0 +1,169 @@

using Data;
using Enums;
namespace Extractors
{
public class StringExtractor : IExtractor
{
public static bool InputMatches(string input)
{
return !input.IsInternetUrl();
}
public async Task<TrackLists> GetTracks()
{
var trackLists = new TrackLists();
var music = ParseTrackArg(Config.input, Config.album);
bool isAlbum = false;
if (Config.album && Config.aggregate)
{
trackLists.AddEntry(ListType.AlbumAggregate, music);
}
else if (Config.album)
{
music.IsAlbum = true;
trackLists.AddEntry(ListType.Album, music);
}
else if (!Config.aggregate && music.Title.Length > 0)
{
trackLists.AddEntry(music);
}
else if (Config.aggregate)
{
trackLists.AddEntry(ListType.Aggregate, music);
}
else if (music.Title.Length == 0 && music.Album.Length > 0)
{
isAlbum = true;
music.IsAlbum = true;
trackLists.AddEntry(ListType.Album, music);
}
else
{
throw new ArgumentException("Need track title or album");
}
if (Config.aggregate || isAlbum || Config.album)
Config.defaultFolderName = music.ToString(true).ReplaceInvalidChars(Config.invalidReplaceStr).Trim();
else
Config.defaultFolderName = ".";
return trackLists;
}
public Track ParseTrackArg(string input, bool isAlbum)
{
input = input.Trim();
var track = new Track();
var keys = new string[] { "title", "artist", "length", "album", "artist-maybe-wrong" };
track.IsAlbum = isAlbum;
void setProperty(string key, string value)
{
switch (key)
{
case "title":
track.Title = value;
break;
case "artist":
track.Artist = value;
break;
case "length":
track.Length = int.Parse(value);
break;
case "album":
track.Album = value;
break;
case "artist-maybe-wrong":
if (value == "true") track.ArtistMaybeWrong = true;
break;
}
}
var parts = input.Split(',');
var other = "";
string? currentKey = null;
string? currentVal = null;
bool otherFieldDone = false;
for (int i = 0; i < parts.Length; i++)
{
var x = parts[i];
bool keyval = false;
if (x.Contains('='))
{
var lr = x.Split('=', 2);
lr[0] = lr[0].Trim();
if (lr.Length == 2 && lr[1].Length > 0 && keys.Contains(lr[0]))
{
if (currentKey != null && currentVal != null)
setProperty(currentKey, currentVal.Trim());
currentKey = lr[0];
currentVal = lr[1];
keyval = true;
otherFieldDone = true;
}
}
if (!keyval && currentVal != null)
{
currentVal += ',' + x;
}
if (!otherFieldDone)
{
if (i > 0) other += ',';
other += x;
}
}
if (currentKey != null && currentVal != null)
setProperty(currentKey, currentVal.Trim());
other = other.Trim();
if (other.Length > 0)
{
string artist = "", album = "", title = "";
parts = other.Split(" - ", StringSplitOptions.TrimEntries | StringSplitOptions.RemoveEmptyEntries);
if (parts.Length == 1 || parts.Length > 3)
{
if (isAlbum)
album = other.Trim();
else
title = other.Trim();
}
else if (parts.Length == 2)
{
artist = parts[0];
if (isAlbum)
album = parts[1];
else
title = parts[1];
}
else if (parts.Length == 3)
{
artist = parts[0];
album = parts[1];
title = parts[2];
}
if (track.Artist.Length == 0)
track.Artist = artist;
if (track.Album.Length == 0)
track.Album = album;
if (track.Title.Length == 0)
track.Title = title;
}
if (track.Title.Length == 0 && track.Album.Length == 0 && track.Artist.Length == 0)
throw new ArgumentException("Track string must contain title, album or artist.");
return track;
}
}
}

675
slsk-batchdl/Extractors/YouTube.cs Normal file
View file

@ -0,0 +1,675 @@
using Google.Apis.YouTube.v3;
using Google.Apis.Services;
using System.Xml;
using YoutubeExplode;
using System.Text.RegularExpressions;
using YoutubeExplode.Common;
using System.Diagnostics;
using HtmlAgilityPack;
using System.Collections.Concurrent;
using Data;
namespace Extractors
{
public class YouTubeExtractor : IExtractor
{
public static bool InputMatches(string input)
{
input = input.ToLower();
return input.IsInternetUrl() && (input.Contains("youtu.be") || input.Contains("youtube.com"));
}
public async Task<TrackLists> GetTracks()
{
var trackLists = new TrackLists();
int max = Config.reverse ? int.MaxValue : Config.maxTracks;
int off = Config.reverse ? 0 : Config.offset;
YouTube.apiKey = Config.ytKey;
string name;
List<Track>? deleted = null;
List<Track> tracks = new();
if (Config.getDeleted)
{
Console.WriteLine("Getting deleted videos..");
var archive = new YouTube.YouTubeArchiveRetriever();
deleted = await archive.RetrieveDeleted(Config.input, printFailed: Config.deletedOnly);
}
if (!Config.deletedOnly)
{
if (YouTube.apiKey.Length > 0)
{
Console.WriteLine("Loading YouTube playlist (API)");
(name, tracks) = await YouTube.GetTracksApi(Config.input, max, off);
}
else
{
Console.WriteLine("Loading YouTube playlist");
(name, tracks) = await YouTube.GetTracksYtExplode(Config.input, max, off);
}
}
else
{
name = await YouTube.GetPlaylistTitle(Config.input);
}
if (deleted != null)
{
tracks.InsertRange(0, deleted);
}
YouTube.StopService();
trackLists.AddEntry(tracks);
if (Config.album || Config.aggregate)
trackLists = TrackLists.FromFlattened(trackLists.Flattened(true, false), Config.aggregate, Config.album);
Config.defaultFolderName = name.ReplaceInvalidChars(Config.invalidReplaceStr);
return trackLists;
}
}
public static class YouTube
{
private static YoutubeClient? youtube = new YoutubeClient();
private static YouTubeService? youtubeService = null;
public static string apiKey = "";
public static async Task<(string, List<Track>)> GetTracksApi(string url, int max = int.MaxValue, int offset = 0)
{
StartService();
string playlistId = await UrlToId(url);
var playlistRequest = youtubeService.Playlists.List("snippet");
playlistRequest.Id = playlistId;
var playlistResponse = playlistRequest.Execute();
string playlistName = playlistResponse.Items[0].Snippet.Title;
var playlistItemsRequest = youtubeService.PlaylistItems.List("snippet,contentDetails");
playlistItemsRequest.PlaylistId = playlistId;
playlistItemsRequest.MaxResults = Math.Min(max, 100);
var tracksDict = await GetDictYtExplode(url, max, offset);
var tracks = new List<Track>();
int count = 0;
while (playlistItemsRequest != null && count < max + offset)
{
var playlistItemsResponse = playlistItemsRequest.Execute();
foreach (var playlistItem in playlistItemsResponse.Items)
{
if (count >= offset)
{
if (tracksDict.ContainsKey(playlistItem.Snippet.ResourceId.VideoId))
tracks.Add(tracksDict[playlistItem.Snippet.ResourceId.VideoId]);
else
{
var title = "";
var uploader = "";
var length = 0;
var desc = "";
var videoRequest = youtubeService.Videos.List("contentDetails,snippet");
videoRequest.Id = playlistItem.Snippet.ResourceId.VideoId;
var videoResponse = videoRequest.Execute();
title = playlistItem.Snippet.Title;
if (videoResponse.Items.Count == 0)
continue;
uploader = videoResponse.Items[0].Snippet.ChannelTitle;
length = (int)XmlConvert.ToTimeSpan(videoResponse.Items[0].ContentDetails.Duration).TotalSeconds;
desc = videoResponse.Items[0].Snippet.Description;
Track track = await ParseTrackInfo(title, uploader, playlistItem.Snippet.ResourceId.VideoId, length, desc);
tracks.Add(track);
}
}
if (++count >= max + offset)
break;
}
if (tracksDict.Count >= 200 && !Console.IsOutputRedirected)
{
Console.SetCursorPosition(0, Console.CursorTop);
Console.Write($"Loaded: {tracks.Count}");
}
playlistItemsRequest.PageToken = playlistItemsResponse.NextPageToken;
if (playlistItemsRequest.PageToken == null || count >= max + offset)
playlistItemsRequest = null;
else
playlistItemsRequest.MaxResults = Math.Min(offset + max - count, 100);
}
Console.WriteLine();
return (playlistName, tracks);
}
// requestInfoIfNeeded=true is way too slow
public static async Task<Track> ParseTrackInfo(string title, string uploader, string id, int length, string desc = "", bool requestInfoIfNeeded = false)
{
(string title, string uploader, int length, string desc) info = ("", "", -1, "");
var track = new Track();
track.URI = id;
uploader = uploader.Replace("–", "-").Trim().RemoveConsecutiveWs();
title = title.Replace("–", "-").Replace(" -- ", " - ").Trim().RemoveConsecutiveWs();
var artist = uploader;
var trackTitle = title;
if (artist.EndsWith(" - Topic"))
{
artist = artist[..^7].Trim();
trackTitle = title;
if (artist == "Various Artists")
{
if (desc.Length == 0 && requestInfoIfNeeded && id.Length > 0)
{
info = await GetVideoInfo(id);
desc = info.desc;
}
if (desc.Length > 0)
{
var lines = desc.Split(new[] { "\r\n", "\r", "\n" }, StringSplitOptions.RemoveEmptyEntries);
var dotLine = lines.FirstOrDefault(line => line.Contains(" · "));
if (dotLine != null)
artist = dotLine.Split(new[] { " · " }, StringSplitOptions.None)[1];
}
}
}
else
{
track.ArtistMaybeWrong = !title.ContainsWithBoundary(artist, true) && !desc.ContainsWithBoundary(artist, true);
var split = title.Split(" - ", StringSplitOptions.TrimEntries | StringSplitOptions.RemoveEmptyEntries);
if (split.Length == 2)
{
artist = split[0];
trackTitle = split[1];
track.ArtistMaybeWrong = false;
}
else if (split.Length > 2)
{
int index = Array.FindIndex(split, s => s.ContainsWithBoundary(artist, true));
if (index != -1 && index < split.Length - 1)
{
artist = split[index];
trackTitle = String.Join(" - ", split[(index + 1)..]);
track.ArtistMaybeWrong = false;
}
}
if (track.ArtistMaybeWrong && requestInfoIfNeeded && desc.Length == 0)
{
info = await GetVideoInfo(id);
track.ArtistMaybeWrong = !info.desc.ContainsWithBoundary(artist, true);
}
}
if (length <= 0 && id.Length > 0 && requestInfoIfNeeded)
{
if (info.length > 0)
length = info.length;
else
{
info = await GetVideoInfo(id);
length = info.length;
}
}
track.Length = length;
track.Artist = artist;
track.Title = trackTitle;
return track;
}
public static async Task<(string title, string uploader, int length, string desc)> GetVideoInfo(string id)
{
(string title, string uploader, int length, string desc) o = ("", "", -1, "");
try
{
var vid = await youtube.Videos.GetAsync(id);
o.title = vid.Title;
o.uploader = vid.Author.ChannelTitle;
o.desc = vid.Description;
o.length = (int)vid.Duration.Value.TotalSeconds;
}
catch
{
if (apiKey.Length > 0)
{
try
{
StartService();
var videoRequest = youtubeService.Videos.List("contentDetails,snippet");
videoRequest.Id = id;
var videoResponse = videoRequest.Execute();
o.title = videoResponse.Items[0].Snippet.Title;
o.uploader = videoResponse.Items[0].Snippet.ChannelTitle;
o.length = (int)XmlConvert.ToTimeSpan(videoResponse.Items[0].ContentDetails.Duration).TotalSeconds;
o.desc = videoResponse.Items[0].Snippet.Description;
}
catch { }
}
}
return o;
}
public static void StartService()
{
if (youtubeService == null)
{
if (apiKey.Length == 0)
throw new Exception("No API key");
youtubeService = new YouTubeService(new BaseClientService.Initializer()
{
ApiKey = apiKey,
ApplicationName = "slsk-batchdl"
});
}
}
public static void StopService()
{
youtubeService = null;
}
public static async Task<Dictionary<string, Track>> GetDictYtExplode(string url, int max = int.MaxValue, int offset = 0)
{
var youtube = new YoutubeClient();
var playlist = await youtube.Playlists.GetAsync(url);
var tracks = new Dictionary<string, Track>();
int count = 0;
await foreach (var video in youtube.Playlists.GetVideosAsync(playlist.Id))
{
if (count >= offset && count < offset + max)
{
var title = video.Title;
var uploader = video.Author.ChannelTitle;
var ytId = video.Id.Value;
var length = (int)video.Duration.Value.TotalSeconds;
var track = await ParseTrackInfo(title, uploader, ytId, length);
tracks[ytId] = track;
}
if (count++ >= offset + max)
break;
}
return tracks;
}
public static async Task<string> GetPlaylistTitle(string url)
{
var youtube = new YoutubeClient();
var playlist = await youtube.Playlists.GetAsync(url);
return playlist.Title;
}
public static async Task<(string, List<Track>)> GetTracksYtExplode(string url, int max = int.MaxValue, int offset = 0)
{
var youtube = new YoutubeClient();
var playlist = await youtube.Playlists.GetAsync(url);
var playlistTitle = playlist.Title;
var tracks = new List<Track>();
int count = 0;
await foreach (var video in youtube.Playlists.GetVideosAsync(playlist.Id))
{
if (count >= offset && count < offset + max)
{
var title = video.Title;
var uploader = video.Author.ChannelTitle;
var ytId = video.Id.Value;
var length = (int)video.Duration.Value.TotalSeconds;
var track = await ParseTrackInfo(title, uploader, ytId, length);
tracks.Add(track);
}
if (count++ >= offset + max)
break;
}
return (playlistTitle, tracks);
}
public static async Task<string> UrlToId(string url)
{
var playlist = await youtube.Playlists.GetAsync(url);
return playlist.Id.ToString();
}
public class YouTubeArchiveRetriever
{
private HttpClient _client;
public YouTubeArchiveRetriever()
{
_client = new HttpClient();
_client.Timeout = TimeSpan.FromSeconds(10);
}
public async Task<List<Track>> RetrieveDeleted(string url, bool printFailed = true)
{
var deletedVideoUrls = new BlockingCollection<string>();
int totalCount = 0;
int archivedCount = 0;
var tracks = new ConcurrentBag<Track>();
var noArchive = new ConcurrentBag<string>();
var failRetrieve = new ConcurrentBag<string>();
int workerCount = 4;
var workers = new List<Task>();
var consoleLock = new object();
void updateInfo()
{
lock (consoleLock)
{
if (!Console.IsOutputRedirected)
{
string info = "Deleted metadata total/archived/retrieved: ";
Console.SetCursorPosition(0, Console.CursorTop);
Console.Write($"{info}{totalCount}/{archivedCount}/{tracks.Count}");
}
}
}
var process = new Process
{
StartInfo = new ProcessStartInfo
{
FileName = "yt-dlp",
Arguments = $"--ignore-no-formats-error --no-warn --match-filter \"!uploader\" --print webpage_url {url}",
RedirectStandardOutput = true,
UseShellExecute = false,
CreateNoWindow = true,
},
EnableRaisingEvents = true
};
process.OutputDataReceived += (sender, e) =>
{
if (!string.IsNullOrWhiteSpace(e.Data))
{
deletedVideoUrls.Add(e.Data);
Interlocked.Increment(ref totalCount);
updateInfo();
}
};
process.Exited += (sender, e) =>
{
deletedVideoUrls.CompleteAdding();
};
process.Start();
process.BeginOutputReadLine();
for (int i = 0; i < workerCount; i++)
{
workers.Add(Task.Run(async () =>
{
foreach (var videoUrl in deletedVideoUrls.GetConsumingEnumerable())
{
var waybackUrls = await GetOldestArchiveUrls(videoUrl, limit: 2);
if (waybackUrls != null && waybackUrls.Count > 0)
{
Interlocked.Increment(ref archivedCount);
bool good = false;
foreach (var waybackUrl in waybackUrls)
{
var (title, uploader, duration) = await GetVideoDetails(waybackUrl);
if (!string.IsNullOrWhiteSpace(title))
{
var track = await ParseTrackInfo(title, uploader, waybackUrl, duration);
track.Other = $"{{\"t\":\"{title.Trim()}\",\"u\":\"{uploader.Trim()}\"}}";
tracks.Add(track);
good = true;
break;
}
}
if (!good)
{
failRetrieve.Add(waybackUrls[0]);
}
}
else
{
noArchive.Add(videoUrl);
}
updateInfo();
}
}));
}
await Task.WhenAll(workers);
process.WaitForExit();
deletedVideoUrls.CompleteAdding();
Console.WriteLine();
if (printFailed)
{
if (archivedCount < totalCount)
{
Console.WriteLine("No archived version found for the following:");
foreach (var x in noArchive)
Console.WriteLine($" {x}");
Console.WriteLine();
}
if (tracks.Count < archivedCount)
{
Console.WriteLine("Failed to parse archived version for the following:");
foreach (var x in failRetrieve)
Console.WriteLine($" {x}");
Console.WriteLine();
}
}
return tracks.ToList();
}
private async Task<List<string>> GetOldestArchiveUrls(string url, int limit)
{
var url2 = $"http://web.archive.org/cdx/search/cdx?url={url}&fl=timestamp,original&filter=statuscode:200&sort=timestamp:asc&limit={limit}";
HttpResponseMessage response = null;
for (int i = 0; i < 3; i++)
{
try
{
response = await _client.GetAsync(url2);
break;
}
catch { }
}
if (response == null) return null;
if (response.IsSuccessStatusCode)
{
var content = await response.Content.ReadAsStringAsync();
var lines = content.Split("\n").Where(line => !string.IsNullOrWhiteSpace(line)).ToList();
if (lines.Count > 0)
{
for (int i = 0; i < lines.Count; i++)
{
var parts = lines[i].Split(" ");
var timestamp = parts[0];
var originalUrl = parts[1];
lines[i] = $"http://web.archive.org/web/{timestamp}/{originalUrl}";
}
return lines;
}
}
return null;
}
public async Task<(string title, string uploader, int duration)> GetVideoDetails(string url)
{
var web = new HtmlWeb();
var doc = await web.LoadFromWebAsync(url);
var titlePatterns = new[]
{
"//h1[@id='video_title']",
"//meta[@name='title']",
};
var usernamePatterns = new[]
{
"//div[@id='userInfoDiv']/b/a",
"//a[contains(@class, 'contributor')]",
"//a[@id='watch-username']",
"//a[contains(@class, 'author')]",
"//div[@class='yt-user-info']/a",
"//div[@id='upload-info']//yt-formatted-string/a",
"//span[@itemprop='author']//link[@itemprop='name']",
"//a[contains(@class, 'yt-user-name')]",
};
string getItem(string[] patterns)
{
foreach (var pattern in patterns)
{
var node = doc.DocumentNode.SelectSingleNode(pattern);
if (node != null)
{
var res = "";
if (pattern.StartsWith("//meta") || pattern.Contains("@itemprop"))
res = node.GetAttributeValue("content", "");
else
res = node.InnerText;
if (!string.IsNullOrEmpty(res))
return Utils.UnHtmlString(res);
}
}
return "";
}
var title = getItem(titlePatterns);
if (string.IsNullOrEmpty(title))
{
var pattern = @"document\.title\s*=\s*""(.+?) - YouTube"";";
var match = Regex.Match(doc.Text, pattern);
if (match.Success)
title = match.Groups[1].Value;
}
var username = getItem(usernamePatterns);
int duration = -1;
var node = doc.DocumentNode.SelectSingleNode("//meta[@itemprop='duration']");
if (node != null)
{
try
{
duration = (int)XmlConvert.ToTimeSpan(node.GetAttributeValue("content", "")).TotalSeconds;
}
catch { }
}
return (title, username, duration);
}
}
public static async Task<List<(int length, string id, string title)>> YtdlpSearch(Track track)
{
Process process = new Process();
ProcessStartInfo startInfo = new ProcessStartInfo();
startInfo.FileName = "yt-dlp";
string search = track.Artist.Length > 0 ? $"{track.Artist} - {track.Title}" : track.Title;
startInfo.Arguments = $"\"ytsearch3:{search}\" --print \"%(duration>%s)s === %(id)s === %(title)s\"";
startInfo.RedirectStandardOutput = true;
startInfo.RedirectStandardError = true;
startInfo.UseShellExecute = false;
process.StartInfo = startInfo;
process.OutputDataReceived += (sender, e) => { Console.WriteLine(e.Data); };
process.ErrorDataReceived += (sender, e) => { Console.WriteLine(e.Data); };
process.Start();
List<(int, string, string)> results = new List<(int, string, string)>();
string output;
Regex regex = new Regex(@"^(\d+) === ([\w-]+) === (.+)$");
while ((output = process.StandardOutput.ReadLine()) != null)
{
Match match = regex.Match(output);
if (match.Success)
{
int seconds = int.Parse(match.Groups[1].Value);
string id = match.Groups[2].Value;
string title = match.Groups[3].Value;
results.Add((seconds, id, title));
}
}
process.WaitForExit();
return results;
}
public static async Task<string> YtdlpDownload(string id, string savePathNoExt, string ytdlpArgument = "")
{
Process process = new Process();
ProcessStartInfo startInfo = new ProcessStartInfo();
if (ytdlpArgument.Length == 0)
ytdlpArgument = "\"{id}\" -f bestaudio/best -ci -o \"{savepath-noext}.%(ext)s\" -x";
startInfo.FileName = "yt-dlp";
startInfo.Arguments = ytdlpArgument
.Replace("{id}", id)
.Replace("{savepath}", savePathNoExt)
.Replace("{savepath-noext}", savePathNoExt)
.Replace("{savedir}", Path.GetDirectoryName(savePathNoExt));
startInfo.RedirectStandardOutput = true;
startInfo.RedirectStandardError = true;
startInfo.UseShellExecute = false;
process.StartInfo = startInfo;
//process.OutputDataReceived += (sender, e) => { Console.WriteLine(e.Data); };
//process.ErrorDataReceived += (sender, e) => { Console.WriteLine(e.Data); };
process.Start();
process.WaitForExit();
if (File.Exists(savePathNoExt + ".opus"))
return savePathNoExt + ".opus";
string parentDirectory = Path.GetDirectoryName(savePathNoExt);
string[] musicFiles = Directory.GetFiles(parentDirectory, "*", SearchOption.TopDirectoryOnly)
.Where(file => Utils.IsMusicFile(file))
.ToArray();
if (musicFiles.Length > 0)
return musicFiles[0];
return "";
}
}
}

36
slsk-batchdl/Extractors/_Extractor.cs Normal file
View file

@ -0,0 +1,36 @@
using Enums;
using Data;
namespace Extractors
{
public interface IExtractor
{
Task<TrackLists> GetTracks();
Task RemoveTrackFromSource(Track track) => Task.CompletedTask;
}
public static class ExtractorRegistry
{
static readonly List<(InputType, Func<string, bool>, Func<IExtractor>)> extractors = new()
{
(InputType.CSV, CsvExtractor.InputMatches, () => new CsvExtractor()),
(InputType.YouTube, YouTubeExtractor.InputMatches, () => new YouTubeExtractor()),
(InputType.Spotify, SpotifyExtractor.InputMatches, () => new SpotifyExtractor()),
(InputType.Bandcamp, BandcampExtractor.InputMatches, () => new BandcampExtractor()),
(InputType.String, StringExtractor.InputMatches, () => new StringExtractor()),
};
public static (InputType, IExtractor?) GetMatchingExtractor(string input)
{
foreach ((var inputType, var inputMatches, var extractor) in extractors)
{
if (inputMatches(input))
{
return (inputType, extractor());
}
}
return (InputType.None, null);
}
}
}

296
slsk-batchdl/FileConditions.cs Normal file
View file

@ -0,0 +1,296 @@
using System.Text.RegularExpressions;
using Data;
using SearchResponse = Soulseek.SearchResponse;
public class FileConditions
{
public int LengthTolerance = -1;
public int MinBitrate = -1;
public int MaxBitrate = -1;
public int MinSampleRate = -1;
public int MaxSampleRate = -1;
public int MinBitDepth = -1;
public int MaxBitDepth = -1;
public bool StrictTitle = false;
public bool StrictArtist = false;
public bool StrictAlbum = false;
public string[] DangerWords = Array.Empty<string>();
public string[] Formats = Array.Empty<string>();
public string[] BannedUsers = Array.Empty<string>();
public string StrictStringRegexRemove = string.Empty;
public bool StrictStringDiacrRemove = true;
public bool AcceptNoLength = true;
public bool AcceptMissingProps = true;
public FileConditions() { }
public FileConditions(FileConditions other)
{
LengthTolerance = other.LengthTolerance;
MinBitrate = other.MinBitrate;
MaxBitrate = other.MaxBitrate;
MinSampleRate = other.MinSampleRate;
MaxSampleRate = other.MaxSampleRate;
AcceptNoLength = other.AcceptNoLength;
StrictArtist = other.StrictArtist;
StrictTitle = other.StrictTitle;
MinBitDepth = other.MinBitDepth;
MaxBitDepth = other.MaxBitDepth;
Formats = other.Formats.ToArray();
DangerWords = other.DangerWords.ToArray();
BannedUsers = other.BannedUsers.ToArray();
}
public override bool Equals(object obj)
{
if (obj is FileConditions other)
{
return LengthTolerance == other.LengthTolerance &&
MinBitrate == other.MinBitrate &&
MaxBitrate == other.MaxBitrate &&
MinSampleRate == other.MinSampleRate &&
MaxSampleRate == other.MaxSampleRate &&
MinBitDepth == other.MinBitDepth &&
MaxBitDepth == other.MaxBitDepth &&
StrictTitle == other.StrictTitle &&
StrictArtist == other.StrictArtist &&
StrictAlbum == other.StrictAlbum &&
StrictStringRegexRemove == other.StrictStringRegexRemove &&
StrictStringDiacrRemove == other.StrictStringDiacrRemove &&
AcceptNoLength == other.AcceptNoLength &&
AcceptMissingProps == other.AcceptMissingProps &&
Formats.SequenceEqual(other.Formats) &&
DangerWords.SequenceEqual(other.DangerWords) &&
BannedUsers.SequenceEqual(other.BannedUsers);
}
return false;
}
public void UnsetClientSpecificFields()
{
MinBitrate = -1;
MaxBitrate = -1;
MinSampleRate = -1;
MaxSampleRate = -1;
MinBitDepth = -1;
MaxBitDepth = -1;
}
public bool FileSatisfies(Soulseek.File file, Track track, SearchResponse? response)
{
return DangerWordSatisfies(file.Filename, track.Title, track.Artist) && FormatSatisfies(file.Filename)
&& LengthToleranceSatisfies(file, track.Length) && BitrateSatisfies(file) && SampleRateSatisfies(file)
&& StrictTitleSatisfies(file.Filename, track.Title) && StrictArtistSatisfies(file.Filename, track.Artist)
&& StrictAlbumSatisfies(file.Filename, track.Album) && BannedUsersSatisfies(response) && BitDepthSatisfies(file);
}
public bool FileSatisfies(TagLib.File file, Track track, bool filenameChecks = false)
{
return DangerWordSatisfies(file.Name, track.Title, track.Artist) && FormatSatisfies(file.Name)
&& LengthToleranceSatisfies(file, track.Length) && BitrateSatisfies(file) && SampleRateSatisfies(file)
&& BitDepthSatisfies(file) && (!filenameChecks || StrictTitleSatisfies(file.Name, track.Title)
&& StrictArtistSatisfies(file.Name, track.Artist) && StrictAlbumSatisfies(file.Name, track.Album));
}
public bool FileSatisfies(SimpleFile file, Track track, bool filenameChecks = false)
{
return DangerWordSatisfies(file.Path, track.Title, track.Artist) && FormatSatisfies(file.Path)
&& LengthToleranceSatisfies(file, track.Length) && BitrateSatisfies(file) && SampleRateSatisfies(file)
&& BitDepthSatisfies(file) && (!filenameChecks || StrictTitleSatisfies(file.Path, track.Title)
&& StrictArtistSatisfies(file.Path, track.Artist) && StrictAlbumSatisfies(file.Path, track.Album));
}
public bool DangerWordSatisfies(string fname, string tname, string aname)
{
if (tname.Length == 0)
return true;
fname = Utils.GetFileNameWithoutExtSlsk(fname).Replace(" — ", " - ");
tname = tname.Replace(" — ", " - ");
foreach (var word in DangerWords)
{
if (fname.ContainsIgnoreCase(word) ^ tname.ContainsIgnoreCase(word))
{
if (!(fname.Contains(" - ") && fname.ContainsIgnoreCase(word) && aname.ContainsIgnoreCase(word)))
{
if (word == "mix")
return fname.ContainsIgnoreCase("original mix") || tname.ContainsIgnoreCase("original mix");
else
return false;
}
}
}
return true;
}
public bool StrictTitleSatisfies(string fname, string tname, bool noPath = true)
{
if (!StrictTitle || tname.Length == 0)
return true;
fname = noPath ? Utils.GetFileNameWithoutExtSlsk(fname) : fname;
return StrictString(fname, tname, StrictStringRegexRemove, StrictStringDiacrRemove, ignoreCase: true);
}
public bool StrictArtistSatisfies(string fname, string aname)
{
if (!StrictArtist || aname.Length == 0)
return true;
return StrictString(fname, aname, StrictStringRegexRemove, StrictStringDiacrRemove, ignoreCase: true, boundarySkipWs: false);
}
public bool StrictAlbumSatisfies(string fname, string alname)
{
if (!StrictAlbum || alname.Length == 0)
return true;
return StrictString(Utils.GetDirectoryNameSlsk(fname), alname, StrictStringRegexRemove, StrictStringDiacrRemove, ignoreCase: true);
}
public static string StrictStringPreprocess(string str, string regexRemove = "", bool diacrRemove = true)
{
str = str.Replace('_', ' ').ReplaceInvalidChars(' ', true, false);
str = regexRemove.Length > 0 ? Regex.Replace(str, regexRemove, "") : str;
str = diacrRemove ? str.RemoveDiacritics() : str;
str = str.Trim().RemoveConsecutiveWs();
return str;
}
public static bool StrictString(string fname, string tname, string regexRemove = "", bool diacrRemove = true, bool ignoreCase = true, bool boundarySkipWs = true)
{
if (tname.Length == 0)
return true;
fname = StrictStringPreprocess(fname, regexRemove, diacrRemove);
tname = StrictStringPreprocess(tname, regexRemove, diacrRemove);
if (boundarySkipWs)
return fname.ContainsWithBoundaryIgnoreWs(tname, ignoreCase, acceptLeftDigit: true);
else
return fname.ContainsWithBoundary(tname, ignoreCase);
}
public static bool BracketCheck(Track track, Track other)
{
string t1 = track.Title.RemoveFt().Replace('[', '(');
if (t1.Contains('('))
return true;
string t2 = other.Title.RemoveFt().Replace('[', '(');
if (!t2.Contains('('))
return true;
return false;
}
public bool FormatSatisfies(string fname)
{
if (Formats.Length == 0)
return true;
string ext = Path.GetExtension(fname).TrimStart('.').ToLower();
return ext.Length > 0 && Formats.Any(f => f == ext);
}
public bool LengthToleranceSatisfies(Soulseek.File file, int wantedLength) => LengthToleranceSatisfies(file.Length, wantedLength);
public bool LengthToleranceSatisfies(TagLib.File file, int wantedLength) => LengthToleranceSatisfies((int)file.Properties.Duration.TotalSeconds, wantedLength);
public bool LengthToleranceSatisfies(SimpleFile file, int wantedLength) => LengthToleranceSatisfies(file.Length, wantedLength);
public bool LengthToleranceSatisfies(int? length, int wantedLength)
{
if (LengthTolerance < 0 || wantedLength < 0)
return true;
if (length == null || length < 0)
return AcceptNoLength && AcceptMissingProps;
return Math.Abs((int)length - wantedLength) <= LengthTolerance;
}
public bool BitrateSatisfies(Soulseek.File file) => BitrateSatisfies(file.BitRate);
public bool BitrateSatisfies(TagLib.File file) => BitrateSatisfies(file.Properties.AudioBitrate);
public bool BitrateSatisfies(SimpleFile file) => BitrateSatisfies(file.Bitrate);
public bool BitrateSatisfies(int? bitrate)
{
return BoundCheck(bitrate, MinBitrate, MaxBitrate);
}
public bool SampleRateSatisfies(Soulseek.File file) => SampleRateSatisfies(file.SampleRate);
public bool SampleRateSatisfies(TagLib.File file) => SampleRateSatisfies(file.Properties.AudioSampleRate);
public bool SampleRateSatisfies(SimpleFile file) => SampleRateSatisfies(file.Samplerate);
public bool SampleRateSatisfies(int? sampleRate)
{
return BoundCheck(sampleRate, MinSampleRate, MaxSampleRate);
}
public bool BitDepthSatisfies(Soulseek.File file) => BitDepthSatisfies(file.BitDepth);
public bool BitDepthSatisfies(TagLib.File file) => BitDepthSatisfies(file.Properties.BitsPerSample);
public bool BitDepthSatisfies(SimpleFile file) => BitDepthSatisfies(file.Bitdepth);
public bool BitDepthSatisfies(int? bitdepth)
{
return BoundCheck(bitdepth, MinBitDepth, MaxBitDepth);
}
public bool BoundCheck(int? num, int min, int max)
{
if (max < 0 && min < 0)
return true;
if (num == null || num < 0)
return AcceptMissingProps;
if (num < min || max != -1 && num > max)
return false;
return true;
}
public bool BannedUsersSatisfies(SearchResponse? response)
{
return response == null || !BannedUsers.Any(x => x == response.Username);
}
public string GetNotSatisfiedName(Soulseek.File file, Track track, SearchResponse? response)
{
if (!DangerWordSatisfies(file.Filename, track.Title, track.Artist))
return "DangerWord fails";
if (!FormatSatisfies(file.Filename))
return "Format fails";
if (!LengthToleranceSatisfies(file, track.Length))
return "Length fails";
if (!BitrateSatisfies(file))
return "Bitrate fails";
if (!SampleRateSatisfies(file))
return "SampleRate fails";
if (!StrictTitleSatisfies(file.Filename, track.Title))
return "StrictTitle fails";
if (!StrictArtistSatisfies(file.Filename, track.Artist))
return "StrictArtist fails";
if (!BitDepthSatisfies(file))
return "BitDepth fails";
if (!BannedUsersSatisfies(response))
return "BannedUsers fails";
return "Satisfied";
}
public string GetNotSatisfiedName(TagLib.File file, Track track)
{
if (!DangerWordSatisfies(file.Name, track.Title, track.Artist))
return "DangerWord fails";
if (!FormatSatisfies(file.Name))
return "Format fails";
if (!LengthToleranceSatisfies(file, track.Length))
return "Length fails";
if (!BitrateSatisfies(file))
return "Bitrate fails";
if (!SampleRateSatisfies(file))
return "SampleRate fails";
if (!StrictTitleSatisfies(file.Name, track.Title))
return "StrictTitle fails";
if (!StrictArtistSatisfies(file.Name, track.Artist))
return "StrictArtist fails";
if (!BitDepthSatisfies(file))
return "BitDepth fails";
return "Satisfied";
}
}

490
slsk-batchdl/Help.cs Normal file
View file

@ -0,0 +1,490 @@

// undocumented options
// --login, --random-login, --no-modify-share-count, --unknown-error-retries
// --invalid-replace-str, --cond, --pref, --danger-words, --pref-danger-words, --strict-title, --strict-artist, --strict-album
// --fast-search-delay, --fast-search-min-up-speed
// --min-album-track-count, --max-album-track-count, --extract-max-track-count, --extract-min-track-count
public static class Help
{
const string helpText = @"
Usage: sldl <input> [OPTIONS]
Required Arguments
<input> A url, search string, or path to a local CSV file.
Run --help ""input"" to view the accepted inputs.
Can also be passed with -i, --input <input>
--user <username> Soulseek username
--pass <password> Soulseek password
General Options
-p, --path <path> Download directory
-f, --folder <name> Subfolder name. Set to '.' to output directly to --path
--input-type <type> Force set input type, [csv|youtube|spotify|bandcamp|string]
--name-format <format> Name format for downloaded tracks. See --help name-format
-n, --number <maxtracks> Download the first n tracks of a playlist
-o, --offset <offset> Skip a specified number of tracks
-r, --reverse Download tracks in reverse order
-c, --config <path> Set config file location. Set to 'none' to ignore config
--profile <name> Configuration profile to use. See --help ""config"".
--concurrent-downloads <num> Max concurrent downloads (default: 2)
--m3u <option> Create an m3u8 playlist file in the output directory
'none' (default for single inputs): Do not create
'index' (default): Write a line indexing all downloaded
files, required for skip-not-found or skip-existing=m3u
'all': Write the index and a list of paths and fails
-s, --skip-existing Skip if a track matching file conditions is found in the
output folder or your music library (if provided)
--skip-mode <mode> [name|tag|m3u|name-cond|tag-cond|m3u-cond]
See --help ""skip-existing"".
--music-dir <path> Specify to also skip downloading tracks found in a music
library. Use with --skip-existing
--skip-not-found Skip searching for tracks that weren't found on Soulseek
during the last run. Fails are read from the m3u file.
--display-mode <option> Changes how searches and downloads are displayed:
'single' (default): Show transfer state and percentage
'double': Transfer state and a large progress bar
'simple': No download bars or changing percentages
--print <option> Print tracks or search results instead of downloading:
'tracks': Print all tracks to be downloaded
'tracks-full': Print extended information about all tracks
'results': Print search results satisfying file conditions
'results-full': Print search results including full paths
--debug Print extra debug info
--listen-port <port> Port for incoming connections (default: 49998)
--on-complete <command> Run a specified command whenever a file is downloaded.
Available placeholders: {path} (local save path), {title},
{artist},{album},{uri},{length},{failure-reason},{state}.
Prepend a state number to only download in specific cases:
1:, 2:, 3:, 4: for the Downloaded, Failed, Exists, and
NotFoundLastTime states respectively.
E.g: '1:<cmd>' will only run the command if the file is
downloaded successfully.
Searching
--fast-search Begin downloading as soon as a file satisfying the preferred
conditions is found. Higher chance to download wrong files.
--remove-ft Remove 'feat.' and everything after before searching
--no-remove-special-chars Do not remove special characters before searching
--remove-brackets Remove square brackets and their contents before searching
--regex <regex> Remove a regexp from all track titles and artist names.
Optionally specify a replacement regex after a semicolon.
Add 'T:', 'A:' or 'L:' at the start to only apply this to
the track title, artist, or album respectively.
--artist-maybe-wrong Performs an additional search without the artist name.
Useful for sources like SoundCloud where the ""artist""
could just be an uploader. Note that when downloading a
YouTube playlist via url, this option is set automatically
on a per-track basis, so it is best kept off in that case.
-d, --desperate Tries harder to find the desired track by searching for the
artist/album/title only, then filtering. (slower search)
--fails-to-downrank <num> Number of fails to downrank a user's uploads (default: 1)
--fails-to-ignore <num> Number of fails to ban/ignore a user's uploads (default: 2)
--yt-dlp Use yt-dlp to download tracks that weren't found on
Soulseek. yt-dlp must be available from the command line.
--yt-dlp-argument <str> The command line arguments when running yt-dlp. Default:
""{id}"" -f bestaudio/best -cix -o ""{savepath}.%(ext)s""
Available vars are: {id}, {savedir}, {savepath} (w/o ext).
Note that with -x, yt-dlp will download webms in case
ffmpeg is unavailable.
--search-timeout <ms> Max search time in ms (default: 6000)
--max-stale-time <ms> Max download time without progress in ms (default: 50000)
--searches-per-time <num> Max searches per time interval. Higher values may cause
30-minute bans, see --help ""search"". (default: 34)
--searches-renew-time <sec> Controls how often available searches are replenished.
See --help ""search"". (default: 220)
Spotify
--spotify-id <id> spotify client ID
--spotify-secret <secret> spotify client secret
--remove-from-source Remove downloaded tracks from source playlist
YouTube
--youtube-key <key> Youtube data API key
--get-deleted Attempt to retrieve titles of deleted videos from wayback
machine. Requires yt-dlp.
--deleted-only Only retrieve & download deleted music.
CSV Files
--artist-col Artist column name
--title-col Track title column name
--album-col Album column name
--length-col Track length column name
--album-track-count-col Album track count column name (sets --album-track-count)
--yt-desc-col Youtube description column (improves --yt-parse)
--yt-id-col Youtube video id column (improves --yt-parse)
--time-format <format> Time format in Length column of the csv file (e.g h:m:s.ms
for durations like 1:04:35.123). Default: s
--yt-parse Enable if the CSV contains YouTube video titles and channel
names; attempt to parse them into title and artist names.
--remove-from-source Remove downloaded tracks from source CSV file
File Conditions
--format <formats> Accepted file format(s), comma-separated, without periods
--length-tol <sec> Length tolerance in seconds
--min-bitrate <rate> Minimum file bitrate
--max-bitrate <rate> Maximum file bitrate
--min-samplerate <rate> Minimum file sample rate
--max-samplerate <rate> Maximum file sample rate
--min-bitdepth <depth> Minimum bit depth
--max-bitdepth <depth> Maximum bit depth
--banned-users <list> Comma-separated list of users to ignore
--pref-format <formats> Preferred file format(s), comma-separated (default: mp3)
--pref-length-tol <sec> Preferred length tolerance in seconds (default: 3)
--pref-min-bitrate <rate> Preferred minimum bitrate (default: 200)
--pref-max-bitrate <rate> Preferred maximum bitrate (default: 2500)
--pref-min-samplerate <rate> Preferred minimum sample rate
--pref-max-samplerate <rate> Preferred maximum sample rate (default: 48000)
--pref-min-bitdepth <depth> Preferred minimum bit depth
--pref-max-bitdepth <depth> Preferred maximum bit depth
--pref-banned-users <list> Comma-separated list of users to downrank
--strict-conditions Skip files with missing properties instead of accepting by
default; if --min-bitrate is set, ignores any files with
unknown bitrate.
Album Download
-a, --album Album download mode
-t, --interactive Allows to select the wanted folder and images
--album-track-count <num> Specify the exact number of tracks in the album. Add a + or
- for inequalities, e.g '5+' for five or more tracks.
--album-ignore-fails Do not skip to the next source and do not delete all
successfully downloaded files if one of the files in the
folder fails to download
--album-art <option> Retrieve additional images after downloading the album:
'default': No additional images
'largest': Download from the folder with the largest image
'most': Download from the folder containing the most images
'most-largest': Do most, then largest
--album-art-only Only download album art for the provided album
--no-browse-folder Do not automatically browse user shares to get all files in
in the folder
Aggregate Download
-g, --aggregate Aggregate download mode: Find and download all distinct
songs associated with the provided artist, album, or title.
--min-users-aggregate <num> Minimum number of users sharing a track or album for it to
be downloaded in aggregate mode. (Default: 2)
--relax-filtering Slightly relax file filtering in aggregate mode to include
more results
Help
-h, --help [option] [all|input|download-modes|search|name-format|
file-conditions|skip-existing|config]
Notes
Acronyms of two- and --three-word-flags are also accepted, e.g. --twf. If the option
contains the word 'max' then the m should be uppercase. 'bitrate', 'sameplerate' and
'bitdepth' should be all treated as two separate words, e.g --Mbr for --max-bitrate.
Flags can be explicitly disabled by setting them to false, e.g '--interactive false'
";
const string inputHelp = @"
Input types
The input type is usually determined automatically. To force a specific input type, set
--input-type [spotify|youtube|csv|string|bandcamp]. The following input types are available:
CSV file
Path to a local CSV file: Use a csv file containing track info of the songs to download.
The names of the columns should be Artist, Title, Album, Length, although alternative names
are usually detected as well. Only the title or album column is required, but extra info may
improve search results. Every row that does not have a title column text will be treated as an
album download.
YouTube
A playlist url: Download songs from a youtube playlist.
The default method to retrieve playlists doesn't always return all videos, especially not
the ones which are unavailable. To get all video titles, you can use the official API by
providing a key with --youtube-key. Get it here https://console.cloud.google.com. Create a
new project, click ""Enable Api"" and search for ""youtube data"", then follow the prompts.
Tip: For playlists containing music videos, it may be better to remove all text in parentheses
(to remove (Lyrics), (Official), etc) and disable song duration checking:
--regex ""[\[\(].*?[\]\)]"" --pref-length-tol -1
Spotify
A playlist/album url or 'spotify-likes': Download a spotify playlist, album, or your
liked songs. --spotify-id and --spotify-secret are required in addition when downloading
a private playlist or liked music.
The id and secret can be obtained at https://developer.spotify.com/dashboard/applications.
Create an app and add http://localhost:48721/callback as a redirect url in its settings.
Bandcamp
An bandcamp url: Download a single track, and album, or an artist's entire discography.
Extracts the artist name, album name and sets --album-track-count=""n+"", where n is the
number of visible tracks on the bandcamp page.
Search string
Name of the track, album, or artist to search for: Can either be any typical search string
(like what you would enter into the soulseek search bar), or a comma-separated list of
properties like 'title=Song Name, artist=Artist Name, length=215'.
The following properties are allowed:
title
artist
album
length (in seconds)
artist-maybe-wrong
Example inputs and their interpretations:
Input String | Artist | Title | Album | Length
---------------------------------------------------------------------------------
'Foo Bar' (without any hyphens) | | Foo Bar | |
'Foo - Bar' | Foo | Bar | |
'Foo - Bar' (with --album enabled) | Foo | | Bar |
'Artist - Title, length=42' | Artist | Title | | 42
'artist=AR, title=T, album=AL' | AR | T | AL |
";
const string downloadModesHelp = @"
Download modes
Normal
The program will download a single file for every input entry.
Album
The program will search for the album and download an entire folder including non-audio
files. Activated when the input is a link to a spotify or bandcamp album, when the input
string or csv row has no track title, or when -a/--album is enabled.
Aggregate
With -g/--aggregate, the program will first perform an ordinary search for the input, then
attempt to group the results into distinct songs and download one of each kind. A common use
case is finding all remixes of a song or printing all songs by an artist that are not your
music dir.
Two files are considered equal if their inferred track title and artist name are equal
(ignoring case and some special characters), and their lengths are within --length-tol of each
other.
Note that this mode is not 100% reliable, which is why --min-users-aggregate is set to 2 by
default, i.e. any song that is shared by only one peer will be ignored. Enable --relax-filtering
to make the file filtering less aggressive.
Album Aggregate
Activated when --album and --aggregate are enabled, in this mode sldl searches for the query
and groups results into distinct albums. Two folders are considered same if they have the
same number of audio files, and the durations of the files are within --length-tol of each
other (or within 3 seconds if length-tol is not configured). If both folders have exactly one
audio file with similar lengths, also checks if the inferred title and artist name coincide.
More reliable than normal aggregate due to much simpler grouping logic.
Note that --min-users-aggregate is 2 by default, which means that folders shared by only one
peer are ignored.
";
const string searchHelp = @"
Searching
Soulseek's rate limits
The server will ban you for 30 minutes if too many searches are performed within a short
timespan. The program has a search limiter which can be adjusted with --searches-per-time
and --searches-renew-time (when the limit is reached, the status of the downloads will be
""Waiting""). By default it is configured to allow up to 34 searches every 220 seconds.
The default values were determined through experimentation, so they may be incorrect.
Quality vs Speed
The following options will make it go faster, but may decrease search result quality or cause
instability:
--fast-search skips waiting until the search completes and downloads as soon as a file
matching the preferred conditions is found
--concurrent-downloads - set it to 4 or more
--max-stale-time is set to 50 seconds by default, so it will wait a long time before giving
up on a file
--searches-per-time increase at the risk of bans.
Quality vs Quantity
The options --strict-title, --strict-artist and --strict-album will filter any file that
does not contain the title/artist/album in the filename (ignoring case, bounded by boundary
chars).
Another way to prevent false downloads is to set --length-tol to 3 or less to make it ignore
any songs that differ from the input by more than 3 seconds. However, all 4 options are already
enabled as 'preferred' conditions by default, meaning that such files will only be downloaded
as a last resort anyways. Hence it is only recommended to enable them if you need to minimize
false downloads as much as possible.
";
const string fileConditionsHelp = @"
File conditions
Files not satisfying the required conditions will not be downloaded. Files satisfying pref-
conditions will be preferred; setting --pref-format ""flac,wav"" will make it download lossless
files if available, and only download lossy files if there's nothing else.
There are no default required conditions. The default preferred conditions are:
format = mp3
length-tol = 3
min-bitrate = 200
max-bitrate = 2500
max-samplerate = 48000
strict-title = true
strict-album = true
accept-no-length = false
sldl will therefore prefer mp3 files with bitrate between 200 and 2500 kbps, and whose length
differs from the supplied length by no more than 3 seconds. It will also prefer files whose
paths contain the supplied artist and album (ignoring case, and bounded by boundary characters)
and which have a non-null length. Changing the last three preferred conditions is not recommended.
Important note
Some info may be unavailable depending on the client used by the peer. For example, the standard
Soulseek client does not share the file bitrate. If (e.g) --min-bitrate is set, then sldl will
still accept any file with unknown bitrate. You can configure it to reject all files where one
or more of the checked properties is null (unknown) by enabling --strict-conditions.
As a consequence, if --min-bitrate is also set then any files shared by users with the default
client will be ignored. Also note that the default preferred conditions will already affect
ranking with this option due to the bitrate and samplerate checks.
Conditions can also be supplied as a semicolon-delimited string with --cond and --pref, e.g
--cond ""br>=320;f=mp3,ogg;sr<96000""
";
const string nameFormatHelp = @"
Name format
Variables enclosed in {} will be replaced by the corresponding file tag value.
Name format supports subdirectories as well as conditional expressions like {tag1|tag2} - If
tag1 is null, use tag2. String literals enclosed in parentheses are ignored in the null check.
Examples:
""{artist} - {title}""
Always name it 'Artist - Title'. Because some files on Soulseek are untagged, the
following is generally preferred:
""{artist( - )title|filename}""
If artist and title are not null, name it 'Artist - Title', otherwise use the original
filename.
Available variables:
artist First artist (from the file tags)
sartist Source artist (as on CSV/Spotify/YouTube/etc)
artists Artists, joined with '&'
albumartist First album artist
albumartists Album artists, joined with '&'
title Track title
stitle Source track title
album Album name
salbum Source album name
year Track year or date
track Track number
disc Disc number
filename Soulseek filename without extension
foldername Default sldl folder name
extractor Name of the extractor used (CSV/Spotify/YouTube/etc)
";
const string skipExistingHelp = @"
Skip existing
sldl can skip files that exist in the download directory or a specified directory configured with
--music-dir.
The following modes are available for --skip-mode:
m3u
Default when checking in the output directory.
Checks whether the output m3u file contains the track in the '#SLDL' line. Does not check if
the audio file exists or satisfies the file conditions, use m3u-cond for that.
name
Compares filenames to the track title and artist name to determine if a track already exists.
Specifically, a track will be skipped if there exists a file whose name contains the title
and whose full path contains the artist name.
tag
Compares file tags to the track title and artist name. A track is skipped if there is a file
whose artist tag contains the track artist and whose title tag equals the track title
(ignoring case and ws). Slower than name mode as it needs to read all file tags.
m3u-cond, name-cond, tag-cond
Default for checking in --music-dir: name-cond.
Same as the above modes but also checks whether the found file satisfies necessary conditions.
Equivalent to the above modes if no necessary conditions have been specified (except m3u-cond
which always checks if the file exists). May be slower and use a lot of memory for large
libraries.
";
const string configHelp = @"
Configuration
Config Location:
sldl will look for a file named sldl.conf in the following locations:
~/AppData/Roaming/sldl/sldl.conf
~/.config/sldl/sldl.conf
as well as in the directory of the executable.
Syntax:
Example config file:
username = your-username
password = your-password
pref-format = flac
fast-search = true
Lines starting with hashtags (#) will be ignored. Tildes in paths are expanded as the user
directory.
Configuration profiles:
Profiles are supported:
[lossless]
pref-format = flac,wav
To activate the above profile, run --profile ""lossless"". To list all available profiles,
run --profile ""help"".
Profiles can be activated automatically based on a few simple conditions:
[no-stale]
profile-cond = interactive && download-mode == ""album""
max-stale-time = 999999
# album downloads will never be automatically cancelled in interactive mode
[youtube]
profile-cond = input-type == ""youtube""
path = ~/downloads/sldl-youtube
# download to another location for youtube
The following operators are supported: &&, ||, ==, !=, ! (negation for bools).
The following variables are available for use in profile-cond:
input-type ( = ""youtube""|""csv""|""string""|""bandcamp""|""spotify"")
download-mode ( = ""normal""|""aggregate""|""album""|""album-aggregate"")
interactive (bool)
";
public static void PrintHelp(string option = "")
{
string text = helpText;
var dict = new Dictionary<string, string>()
{
{ "input", inputHelp },
{ "download-modes", downloadModesHelp },
{ "search", searchHelp },
{ "file-conditions", fileConditionsHelp },
{ "name-format", nameFormatHelp },
{ "skip-existing", skipExistingHelp },
{ "config", configHelp },
};
if (dict.ContainsKey(option))
text = dict[option];
else if (option == "all")
text = $"{helpText}\n{string.Join('\n', dict.Values)}";
else if (option.Length > 0)
Console.WriteLine($"Unrecognized help option '{option}'");
var lines = text.Split('\n').Skip(1);
int minIndent = lines.Where(line => line.Trim().Length > 0).Min(line => line.TakeWhile(char.IsWhiteSpace).Count());
text = string.Join("\n", lines.Select(line => line.Length > minIndent ? line[minIndent..] : line));
Console.WriteLine(text);
}
}

299
slsk-batchdl/M3uEditor.cs Normal file
View file

@ -0,0 +1,299 @@
using Data;
using Enums;
using System.Text;
public class M3uEditor
{
List<string> lines;
TrackLists trackLists;
string path;
string parent;
int offset = 0;
M3uOption option = M3uOption.Index;
bool needFirstUpdate = false;
Dictionary<string, Track> previousRunTracks = new(); // {track.ToKey(), track }
public M3uEditor(string m3uPath, TrackLists trackLists, M3uOption option, int offset = 0)
{
this.trackLists = trackLists;
this.offset = offset;
this.option = option;
this.path = Path.GetFullPath(m3uPath);
this.parent = Path.GetDirectoryName(path);
this.lines = ReadAllLines().ToList();
this.needFirstUpdate = option == M3uOption.All;
LoadPreviousResults();
}
private void LoadPreviousResults() // #SLDL:path,artist,album,title,length(int),state(int),failurereason(int); ... ; ...
{
if (lines.Count == 0 || !lines[0].StartsWith("#SLDL:"))
return;
string sldlLine = lines[0]["#SLDL:".Length..];
var currentItem = new StringBuilder();
bool inQuotes = false;
lines = lines.Skip(1).ToList();
for (int k = 0; k < sldlLine.Length; k++)
{
var track = new Track();
int field = 0;
for (int i = k; i < sldlLine.Length; i++)
{
char c = sldlLine[i];
if (c == '"' && (i == k || sldlLine[i - 1] != '\\'))
{
if (inQuotes && i + 1 < sldlLine.Length && sldlLine[i + 1] == '"')
{
currentItem.Append('"');
i++;
}
else
{
inQuotes = !inQuotes;
}
}
else if (field <= 5 && c == ',' && !inQuotes)
{
var x = currentItem.ToString();
if (field == 0)
{
if (x.StartsWith("./"))
x = Path.Join(parent, x[2..]);
track.DownloadPath = x;
}
else if (field == 1)
track.Artist = x;
else if (field == 2)
track.Album = x;
else if (field == 3)
track.Title = x;
else if (field == 4)
track.Length = int.Parse(x);
else if (field == 5)
track.State = (TrackState)int.Parse(x);
currentItem.Clear();
field++;
}
else if (field == 6 && c == ';')
{
track.FailureReason = (FailureReason)int.Parse(currentItem.ToString());
currentItem.Clear();
k = i;
break;
}
else
{
currentItem.Append(c);
}
}
previousRunTracks[track.ToKey()] = track;
}
}
public void Update()
{
if (option == M3uOption.None)
return;
lock (trackLists)
{
bool needUpdate = false;
int index = 1 + offset;
void updateLine(string newLine)
{
while (index >= lines.Count) lines.Add("");
lines[index] = newLine;
}
foreach (var tle in trackLists.lists)
{
if (tle.type != ListType.Normal)
{
continue;
}
//if (option == M3uOption.All && source.State == TrackState.Failed)
//{
// string reason = source.FailureReason.ToString();
// updateLine(TrackToLine(source, reason));
// index++;
//}
else
{
for (int k = 0; k < tle.list.Count; k++)
{
for (int j = 0; j < tle.list[k].Count; j++)
{
var track = tle.list[k][j];
if (track.IsNotAudio || track.State == TrackState.Initial)
continue;
string trackKey = track.ToKey();
previousRunTracks.TryGetValue(trackKey, out Track? indexTrack);
if (!needUpdate)
{
needUpdate |= indexTrack == null
|| indexTrack.State != track.State
|| indexTrack.FailureReason != track.FailureReason
|| indexTrack.DownloadPath != track.DownloadPath;
}
previousRunTracks[trackKey] = track;
if (option == M3uOption.All)
{
if (track.State != TrackState.AlreadyExists || k == 0)
{
string? reason = track.FailureReason != FailureReason.None ? track.FailureReason.ToString() : null;
if (reason == null && track.State == TrackState.NotFoundLastTime)
reason = nameof(FailureReason.NoSuitableFileFound);
updateLine(TrackToLine(track, reason));
if (tle.type != ListType.Normal)
index++;
}
}
if (tle.type == ListType.Normal)
index++;
}
}
}
}
if (needUpdate || needFirstUpdate)
{
needFirstUpdate = false;
WriteAllLines();
}
}
}
private void WriteAllLines()
{
if (!Directory.Exists(parent))
Directory.CreateDirectory(parent);
using var fileStream = new FileStream(path, FileMode.OpenOrCreate, FileAccess.Write, FileShare.ReadWrite);
using var writer = new StreamWriter(fileStream);
WriteSldlLine(writer);
foreach (var line in lines)
{
writer.Write(line);
writer.Write('\n');
}
}
private void WriteSldlLine(StreamWriter writer) // #SLDL:path,artist,album,title,length(int),state(int),failurereason(int); ... ; ...
{
void writeCsvLine(string[] items)
{
bool comma = false;
foreach (var item in items)
{
if (comma)
writer.Write(',');
if (item.Contains(',') || item.Contains('\"'))
{
writer.Write('"');
writer.Write(item.Replace("\"", "\"\""));
writer.Write('"');
}
else
{
writer.Write(item);
}
comma = true;
}
}
writer.Write("#SLDL:");
foreach (var val in previousRunTracks.Values)
{
string p = val.DownloadPath;
if (p.StartsWith(parent))
p = "./" + Path.GetRelativePath(parent, p); // prepend ./ for LoadPreviousResults to recognize that a rel. path is used
var items = new string[]
{
p,
val.Artist,
val.Album,
val.Title,
val.Length.ToString(),
((int)val.State).ToString(),
((int)val.FailureReason).ToString(),
};
writeCsvLine(items);
writer.Write(';');
}
writer.Write('\n');
}
private string TrackToLine(Track track, string? failureReason = null)
{
if (failureReason != null)
return $"# Failed: {track} [{failureReason}]";
if (track.DownloadPath.Length > 0)
{
if (track.DownloadPath.StartsWith(parent))
return Path.GetRelativePath(parent, track.DownloadPath);
else
return track.DownloadPath;
}
return $"# {track}";
}
public Track? PreviousRunResult(Track track)
{
previousRunTracks.TryGetValue(track.ToKey(), out var t);
return t;
}
public bool TryGetPreviousRunResult(Track track, out Track? result)
{
previousRunTracks.TryGetValue(track.ToKey(), out result);
return result != null;
}
public bool TryGetFailureReason(Track track, out FailureReason reason)
{
reason = FailureReason.None;
var t = PreviousRunResult(track);
if (t != null && t.State == TrackState.Failed)
{
reason = t.FailureReason;
return true;
}
return false;
}
private string ReadAllText()
{
if (!File.Exists(path))
return "";
using var fileStream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
using var streamReader = new StreamReader(fileStream);
return streamReader.ReadToEnd();
}
private string[] ReadAllLines()
{
return ReadAllText().TrimEnd().Split('\n');
}
}

4536
slsk-batchdl/Program.cs
View file

File diff suppressed because it is too large Load diff

1338
slsk-batchdl/SearchAndDownload.cs Normal file
View file

File diff suppressed because it is too large Load diff

220
slsk-batchdl/Spotify.cs
View file

@ -1,220 +0,0 @@
using SpotifyAPI.Web;
using SpotifyAPI.Web.Auth;
using Swan;
public class Spotify
{
private EmbedIOAuthServer _server;
private readonly string _clientId;
private readonly string _clientSecret;
private SpotifyClient? _client;
private bool loggedIn = false;
// default spotify credentials (base64-encoded to keep the bots away)
public const string encodedSpotifyId = "MWJmNDY5M1bLaH9WJiYjFhNGY0MWJjZWQ5YjJjMWNmZGJiZDI=";
public const string encodedSpotifySecret = "Y2JlM2QxYTE5MzJkNDQ2MmFiOGUy3shTuf4Y2JhY2M3ZDdjYWU=";
public bool UsedDefaultCredentials { get; private set; }
public Spotify(string clientId="", string clientSecret="")
{
_clientId = clientId;
_clientSecret = clientSecret;
if (_clientId == "" || _clientSecret == "")
{
_clientId = System.Text.Encoding.UTF8.GetString(Convert.FromBase64String(encodedSpotifyId.Replace("1bLaH9", "")));
_clientSecret = System.Text.Encoding.UTF8.GetString(Convert.FromBase64String(encodedSpotifySecret.Replace("3shTuf4", "")));
UsedDefaultCredentials = true;
}
}
public async Task Authorize(bool login = false, bool needModify = false)
{
_client = null;
if (!login)
{
var config = SpotifyClientConfig.CreateDefault();
var request = new ClientCredentialsRequest(_clientId, _clientSecret);
var response = await new OAuthClient(config).RequestToken(request);
_client = new SpotifyClient(config.WithToken(response.AccessToken));
}
else
{
Swan.Logging.Logger.NoLogging();
_server = new EmbedIOAuthServer(new Uri("http://localhost:48721/callback"), 48721);
await _server.Start();
_server.AuthorizationCodeReceived += OnAuthorizationCodeReceived;
_server.ErrorReceived += OnErrorReceived;
var scope = new List<string> {
Scopes.UserLibraryRead, Scopes.PlaylistReadPrivate, Scopes.PlaylistReadCollaborative
};
if (needModify)
{
scope.Add(Scopes.PlaylistModifyPublic);
scope.Add(Scopes.PlaylistModifyPrivate);
}
var request = new LoginRequest(_server.BaseUri, _clientId, LoginRequest.ResponseType.Code) { Scope = scope };
BrowserUtil.Open(request.ToUri());
await IsClientReady();
}
}
private async Task OnAuthorizationCodeReceived(object sender, AuthorizationCodeResponse response)
{
await _server.Stop();
var config = SpotifyClientConfig.CreateDefault();
var tokenResponse = await new OAuthClient(config).RequestToken(
new AuthorizationCodeTokenRequest(
_clientId, _clientSecret, response.Code, new Uri("http://localhost:48721/callback")
)
);
_client = new SpotifyClient(tokenResponse.AccessToken);
loggedIn = true;
}
private async Task OnErrorReceived(object sender, string error, string state)
{
await _server.Stop();
throw new Exception($"Aborting authorization, error received: {error}");
}
public async Task<bool> IsClientReady()
{
while (_client == null)
await Task.Delay(1000);
return true;
}
public async Task<List<Track>> GetLikes(int max = int.MaxValue, int offset = 0)
{
if (!loggedIn)
throw new Exception("Can't get liked music as user is not logged in");
List<Track> res = new List<Track>();
int limit = Math.Min(max, 50);
while (true)
{
var tracks = await _client.Library.GetTracks(new LibraryTracksRequest { Limit = limit, Offset = offset });
foreach (var track in tracks.Items)
{
string[] artists = ((IEnumerable<object>)track.Track.ReadProperty("artists")).Select(a => (string)a.ReadProperty("name")).ToArray();
string artist = artists[0];
string name = (string)track.Track.ReadProperty("name");
string album = (string)track.Track.ReadProperty("album").ReadProperty("name");
int duration = (int)track.Track.ReadProperty("durationMs");
res.Add(new Track { Album = album, Artist = artist, Title = name, Length = duration / 1000 });
}
if (tracks.Items.Count < limit || res.Count >= max)
break;
offset += limit;
limit = Math.Min(max - res.Count, 50);
}
return res;
}
public async Task RemoveTrackFromPlaylist(string playlistId, string trackUri)
{
var item = new PlaylistRemoveItemsRequest.Item { Uri = trackUri };
var pr = new PlaylistRemoveItemsRequest();
pr.Tracks = new List<PlaylistRemoveItemsRequest.Item>() { item };
try { await _client.Playlists.RemoveItems(playlistId, pr); }
catch { }
}
public async Task<(string?, string?, List<Track>)> GetPlaylist(string url, int max = int.MaxValue, int offset = 0)
{
var playlistId = GetPlaylistIdFromUrl(url);
var p = await _client.Playlists.Get(playlistId);
List<Track> res = new List<Track>();
int limit = Math.Min(max, 100);
while (true)
{
var tracks = await _client.Playlists.GetItems(playlistId, new PlaylistGetItemsRequest { Limit = limit, Offset = offset });
foreach (var track in tracks.Items)
{
try
{
string[] artists = ((IEnumerable<object>)track.Track.ReadProperty("artists")).Select(a => (string)a.ReadProperty("name")).ToArray();
var t = new Track()
{
Artist = artists[0],
Album = (string)track.Track.ReadProperty("album").ReadProperty("name"),
Title = (string)track.Track.ReadProperty("name"),
Length = (int)track.Track.ReadProperty("durationMs") / 1000,
URI = (string)track.Track.ReadProperty("uri"),
};
res.Add(t);
}
catch
{
continue;
}
}
if (tracks.Items.Count < limit || res.Count >= max)
break;
offset += limit;
limit = Math.Min(max - res.Count, 100);
}
return (p.Name, p.Id, res);
}
private string GetPlaylistIdFromUrl(string url)
{
var uri = new Uri(url);
var segments = uri.Segments;
return segments[segments.Length - 1].TrimEnd('/');
}
public async Task<(Track, List<Track>)> GetAlbum(string url)
{
var albumId = GetAlbumIdFromUrl(url);
var album = await _client.Albums.Get(albumId);
List<Track> tracks = new List<Track>();
foreach (var track in album.Tracks.Items)
{
var t = new Track()
{
Album = album.Name,
Artist = track.Artists.First().Name,
Title = track.Name,
Length = track.DurationMs / 1000,
URI = track.Uri,
};
tracks.Add(t);
}
return (new Track { Album = album.Name, Artist = album.Artists.First().Name, IsAlbum = true }, tracks);
}
private string GetAlbumIdFromUrl(string url)
{
var uri = new Uri(url);
var segments = uri.Segments;
return segments[segments.Length - 1].TrimEnd('/');
}
}

643
slsk-batchdl/Utils.cs
View file

@ -1,14 +1,16 @@
using System.Net;
using System.Text;
using System.Text.RegularExpressions;
using Enums;
public static class Utils
{
public static string[] musicExtensions = new string[] { ".mp3", ".wav", ".flac", ".ogg", ".aac", ".wma", ".m4a", ".alac", ".ape", ".opus" };
public static string[] imageExtensions = new string[] { ".jpg", ".jpeg", ".png" };
public static readonly string[] musicExtensions = new string[] { ".mp3", ".flac", ".ogg", ".m4a", ".opus", ".wav", ".aac", ".alac" };
public static readonly string[] imageExtensions = new string[] { ".jpg", ".png", ".jpeg" };
public static bool IsMusicExtension(string extension)
{
return musicExtensions.Contains(('.' + extension.TrimStart('.')).ToLower());
return musicExtensions.Contains(extension.ToLower());
}
public static bool IsMusicFile(string fileName)
@ -16,11 +18,123 @@ public static class Utils
return musicExtensions.Contains(Path.GetExtension(fileName).ToLower());
}
public static bool IsImageExtension(string extension)
{
return imageExtensions.Contains(extension.ToLower());
}
public static bool IsImageFile(string fileName)
{
return imageExtensions.Contains(Path.GetExtension(fileName).ToLower());
}
public static bool IsInternetUrl(this string str)
{
str = str.TrimStart();
return str.StartsWith("http://", StringComparison.OrdinalIgnoreCase)
|| str.StartsWith("https://", StringComparison.OrdinalIgnoreCase);
}
public static void WriteAllLines(string path, IEnumerable<string> lines, char separator)
{
using (var writer = new StreamWriter(path))
{
foreach (var line in lines)
{
writer.Write(line);
writer.Write(separator);
}
}
}
public static string GetAsPathSlsk(string fname)
{
return fname.Replace('\\', Path.DirectorySeparatorChar);
}
public static string GetFileNameSlsk(string fname)
{
fname = fname.Replace('\\', Path.DirectorySeparatorChar);
return Path.GetFileName(fname);
}
public static string GetBaseNameSlsk(string path)
{
path = path.Replace('\\', Path.DirectorySeparatorChar).TrimEnd(Path.DirectorySeparatorChar);
return Path.GetFileName(path);
}
public static string GetFileNameWithoutExtSlsk(string fname)
{
fname = fname.Replace('\\', Path.DirectorySeparatorChar);
return Path.GetFileNameWithoutExtension(fname);
}
public static string GetExtensionSlsk(string fname)
{
fname = fname.Replace('\\', Path.DirectorySeparatorChar);
return Path.GetExtension(fname);
}
public static string GetDirectoryNameSlsk(string fname)
{
fname = fname.Replace('\\', Path.DirectorySeparatorChar);
return Path.GetDirectoryName(fname);
}
public static string ExpandUser(string path)
{
if (string.IsNullOrEmpty(path))
{
return path;
}
if (path.StartsWith("~"))
{
string homeDirectory = Environment.GetFolderPath(Environment.SpecialFolder.UserProfile);
return Path.Combine(homeDirectory, path.Substring(1).TrimStart('/').TrimStart('\\'));
}
return path;
}
public static List<string> FromCsv(string csvLine)
{
var items = new List<string>();
var currentItem = new StringBuilder();
bool inQuotes = false;
for (int i = 0; i < csvLine.Length; i++)
{
char c = csvLine[i];
if (c == '"' && (i == 0 || csvLine[i - 1] != '\\'))
{
if (inQuotes && i + 1 < csvLine.Length && csvLine[i + 1] == '"')
{
currentItem.Append('"');
i++;
}
else
{
inQuotes = !inQuotes;
}
}
else if (c == ',' && !inQuotes)
{
items.Add(currentItem.ToString());
currentItem.Clear();
}
else
{
currentItem.Append(c);
}
}
items.Add(currentItem.ToString());
return items;
}
public static decimal Normalize(this double value)
{
return ((decimal)value) / 1.000000000000000000000000000000000m;
@ -60,9 +174,13 @@ public static class Utils
public static string Replace(this string s, string[] separators, string newVal)
{
string[] temp;
temp = s.Split(separators, StringSplitOptions.RemoveEmptyEntries);
return String.Join(newVal, temp);
if (s.Length == 0)
return s;
foreach (var sep in separators)
s = s.Replace(sep, newVal);
return s;
}
public static string UnHtmlString(this string s)
@ -76,117 +194,233 @@ public static class Utils
return s;
}
public static string ReplaceInvalidChars(this string str, string replaceStr, bool windows = false, bool removeSlash = true, bool alwaysRemoveSlash = false)
public static string ReplaceInvalidChars(this string str, string replaceStr, bool windows = false, bool removeSlash = true)
{
char[] invalidChars = Path.GetInvalidFileNameChars();
if (str.Length == 0)
return str;
char[] invalidChars;
if (windows)
invalidChars = new char[] { ':', '|', '?', '>', '<', '*', '"', '/', '\\' };
if (!removeSlash && !alwaysRemoveSlash)
invalidChars = invalidChars.Where(c => c != '/' && c != '\\').ToArray();
if (alwaysRemoveSlash)
invalidChars = new char[] { ':', '|', '?', '>', '<', '*', '"' }; // forward- and backslash are always included
else
invalidChars = Path.GetInvalidFileNameChars();
if (removeSlash)
{
var x = invalidChars.ToList();
x.AddRange(new char[] { '/', '\\' });
invalidChars = x.ToArray();
str = str.Replace("/", replaceStr);
str = str.Replace("\\", replaceStr);
}
foreach (char c in invalidChars)
foreach (var c in invalidChars)
{
if (!removeSlash && (c == '/' || c == '\\'))
continue;
str = str.Replace(c.ToString(), replaceStr);
}
return str;
}
public static string ReplaceInvalidChars(this string str, char replaceChar, bool windows = false, bool removeSlash = true)
{
if (str.Length == 0)
return str;
char[] invalidChars;
if (windows)
invalidChars = new char[] { ':', '|', '?', '>', '<', '*', '"' }; // forward- and backslash are always included
else
invalidChars = Path.GetInvalidFileNameChars();
if (removeSlash)
{
str = str.Replace('/', replaceChar);
str = str.Replace('\\', replaceChar);
}
foreach (var c in invalidChars)
{
if (!removeSlash && (c == '/' || c == '\\'))
continue;
str = str.Replace(c, replaceChar);
}
return str;
}
public static string ReplaceSpecialChars(this string str, string replaceStr)
{
if (str.Length == 0)
return str;
string special = ";:'\"|?!<>*/\\[]{}()-–—&%^$#@+=`~_";
foreach (char c in special)
str = str.Replace(c.ToString(), replaceStr);
return str;
}
public static string RemoveFt(this string str, bool removeParentheses = true, bool onlyIfNonempty = true)
public static string RemoveFt(this string str, bool removeParentheses = true)
{
string[] ftStrings = { "feat.", "ft." };
string orig = str;
if (str.Length == 0)
return str;
var ftStrings = new string[] { "feat.", "ft." };
var open = new char[] { '(', '[' };
var close = new char[] { ')', ']' };
bool changed = false;
foreach (string ftStr in ftStrings)
{
int ftIndex = str.IndexOf(ftStr, StringComparison.OrdinalIgnoreCase);
if (ftIndex != -1)
{
if (removeParentheses)
changed = true;
if (removeParentheses && ftIndex > 0)
{
int openingParenthesesIndex = str.LastIndexOf('(', ftIndex);
int closingParenthesesIndex = str.IndexOf(')', ftIndex);
int openingBracketIndex = str.LastIndexOf('[', ftIndex);
int closingBracketIndex = str.IndexOf(']', ftIndex);
bool any = false;
if (openingParenthesesIndex != -1 && closingParenthesesIndex != -1)
str = str.Remove(openingParenthesesIndex, closingParenthesesIndex - openingParenthesesIndex + 1);
else if (openingBracketIndex != -1 && closingBracketIndex != -1)
str = str.Remove(openingBracketIndex, closingBracketIndex - openingBracketIndex + 1);
else
str = str.Substring(0, ftIndex);
for (int i = 0; i < 2; i++)
{
if (str[ftIndex - 1] == open[i])
{
int openIdx = ftIndex - 1;
int closeIdx = str.IndexOf(close[i], ftIndex);
if (closeIdx != -1)
{
int add = 0;
if (openIdx > 0 && closeIdx < str.Length - 1 && str[openIdx - 1] == ' ' && str[closeIdx + 1] == ' ')
add = 1;
str = str.Remove(openIdx, closeIdx - openIdx + 1 + add);
any = true;
break;
}
}
}
if (!any)
{
str = str[..ftIndex];
}
}
else
str = str.Substring(0, ftIndex);
{
str = str[..ftIndex];
}
break;
}
}
if (onlyIfNonempty)
str = str.TrimEnd() == "" ? orig : str;
return str.TrimEnd();
return changed ? str.TrimEnd() : str;
}
public static string RemoveConsecutiveWs(this string input)
{
return string.Join(' ', input.Split(new char[] { ' ' }, StringSplitOptions.RemoveEmptyEntries));
if (input.Length == 0)
return string.Empty;
int index = 0;
var src = input.ToCharArray();
bool skip = false;
char ch;
for (int i = 0; i < input.Length; i++)
{
ch = src[i];
if (ch == ' ')
{
if (skip) continue;
src[index++] = ch;
skip = true;
}
else
{
skip = false;
src[index++] = ch;
}
}
return new string(src, 0, index);
}
public static string RemoveSquareBrackets(this string str)
{
if (str.Length == 0)
return str;
if (!str.Contains('['))
return str;
return Regex.Replace(str, @"\[[^\]]*\]", "").Trim();
}
public static bool RemoveDiacriticsIfExist(this string s, out string res)
{
res = s.RemoveDiacritics();
return res != s;
}
public static bool ContainsIgnoreCase(this string s, string other)
{
return s.Contains(other, StringComparison.OrdinalIgnoreCase);
}
static char[] boundaryChars = { '-', '|', '.', '\\', '/', '_', '—', '(', ')', '[', ']', ',', '?', '!', ';',
'@', ':', '*', '=', '+', '{', '}', '|', '\'', '"', '$', '^', '&', '#', '`', '~', '%', '<', '>' };
static string boundaryPattern = "^|$|" + string.Join("|", boundaryChars.Select(c => Regex.Escape(c.ToString())));
static readonly HashSet<char> boundarySet = new("-|.\\/_—()[],:?!;@:*=+{}|'\"$^&`~%<>".ToCharArray());
public static bool ContainsWithBoundary(this string str, string value, bool ignoreCase = false)
{
if (value == "")
if (value.Length == 0)
return true;
if (str == "")
if (str.Length == 0)
return false;
string bound = boundaryPattern + "|\\s";
string pattern = $@"({bound}){Regex.Escape(value)}({bound})";
RegexOptions options = ignoreCase ? RegexOptions.IgnoreCase : RegexOptions.None;
return Regex.IsMatch(str, pattern, options);
var comp = ignoreCase ? StringComparison.OrdinalIgnoreCase : StringComparison.Ordinal;
int index = 0;
while ((index = str.IndexOf(value, index, comp)) != -1)
{
bool hasLeftBoundary = index == 0 || str[index - 1] == ' ' || boundarySet.Contains(str[index - 1]);
bool hasRightBoundary = index + value.Length >= str.Length || str[index + value.Length] == ' ' || boundarySet.Contains(str[index + value.Length]);
if (hasLeftBoundary && hasRightBoundary)
return true;
index += value.Length;
}
return false;
}
public static bool ContainsWithBoundaryIgnoreWs(this string str, string value, bool ignoreCase = false, bool acceptLeftDigit = false)
{
if (value == "")
if (value.Length == 0)
return true;
if (str == "")
if (str.Length == 0)
return false;
string patternLeft = acceptLeftDigit ? boundaryPattern + @"|\d\s+" : boundaryPattern;
string pattern = $@"({patternLeft})\s*{Regex.Escape(value)}\s*({boundaryPattern})";
RegexOptions options = ignoreCase ? RegexOptions.IgnoreCase : RegexOptions.None;
return Regex.IsMatch(str, pattern, options);
}
var comp = ignoreCase ? StringComparison.OrdinalIgnoreCase : StringComparison.Ordinal;
int index = 0;
while ((index = str.IndexOf(value, index, comp)) != -1)
{
int leftIndex = index - 1;
while (leftIndex >= 0 && str[leftIndex] == ' ')
leftIndex--;
bool hasLeftBoundary = leftIndex < 0 || acceptLeftDigit && leftIndex < index - 1 && char.IsDigit(str[leftIndex]) || boundarySet.Contains(str[leftIndex]);
int rightIndex = index + value.Length;
while (rightIndex < str.Length && str[rightIndex] == ' ')
rightIndex++;
bool hasRightBoundary = rightIndex >= str.Length || boundarySet.Contains(str[rightIndex]);
if (hasLeftBoundary && hasRightBoundary)
return true;
index = rightIndex;
}
return false;
}
public static bool ContainsInBrackets(this string str, string searchTerm, bool ignoreCase = false)
{
if (str.Length == 0 && searchTerm.Length > 0)
return false;
var regex = new Regex(@"\[(.*?)\]|\((.*?)\)");
var matches = regex.Matches(str);
var comp = ignoreCase ? StringComparison.OrdinalIgnoreCase : StringComparison.Ordinal;
@ -206,16 +440,6 @@ public static class Utils
return res != s;
}
public static char RemoveDiacritics(this char c)
{
foreach (var entry in diacriticChars)
{
if (entry.Key.IndexOf(c) != -1)
return entry.Value[0];
}
return c;
}
public static Dictionary<K, V> ToSafeDictionary<T, K, V>(this IEnumerable<T> source, Func<T, K> keySelector, Func<T, V> valSelector)
{
var d = new Dictionary<K, V>();
@ -256,113 +480,222 @@ public static class Utils
return distance[source.Length, target.Length];
}
public static string GreatestCommonPath(IEnumerable<string> paths, char dirsep = '-')
public static string GreatestCommonPath(IEnumerable<string> paths, char dirsep)
{
var commonPath = paths.FirstOrDefault();
if (string.IsNullOrEmpty(commonPath))
string? path = paths.FirstOrDefault();
if (path == null || path.Length == 0)
return "";
foreach (var path in paths.Skip(1))
int commonPathIndex(string path1, string path2, int maxIndex)
{
commonPath = GetCommonPath(commonPath, path, dirsep);
var minLength = Math.Min(path1.Length, Math.Min(path2.Length, maxIndex));
var commonPathLength = 0;
for (int i = 0; i < minLength; i++)
{
if (path1[i] != path2[i])
break;
if (path1[i] == dirsep)
commonPathLength = i + 1;
}
return commonPathLength;
}
return commonPath;
int index = path.Length;
foreach (var p in paths.Skip(1))
index = commonPathIndex(path, p, index);
return path[..index];
}
private static string GetCommonPath(string path1, string path2, char dirsep = '-')
public static bool SequenceEqualUpToPermutation<T>(this IEnumerable<T> list1, IEnumerable<T> list2)
{
if (dirsep == '-')
dirsep = Path.DirectorySeparatorChar;
var minLength = Math.Min(path1.Length, path2.Length);
var commonPathLength = 0;
for (int i = 0; i < minLength; i++)
var cnt = new Dictionary<T, int>();
foreach (T s in list1)
{
if (path1[i] != path2[i])
break;
if (path1[i] == dirsep)
commonPathLength = i + 1;
if (cnt.ContainsKey(s))
cnt[s]++;
else
cnt.Add(s, 1);
}
return path1.Substring(0, commonPathLength);
foreach (T s in list2)
{
if (cnt.ContainsKey(s))
cnt[s]--;
else
return false;
}
return cnt.Values.All(c => c == 0);
}
public static bool RemoveDiacriticsIfExist(this string s, out string res)
{
res = s.RemoveDiacritics();
return res != s;
}
public static char RemoveDiacritics(this char c)
{
if (diacriticChars.TryGetValue(c, out var res)) return res;
return c;
}
public static string RemoveDiacritics(this string s)
{
string text = "";
if (s.Length == 0)
return s;
var textBuilder = new StringBuilder();
foreach (char c in s)
{
int len = text.Length;
foreach (var entry in diacriticChars)
{
if (entry.Key.IndexOf(c) != -1)
{
text += entry.Value;
break;
}
}
if (len == text.Length)
text += c;
if (diacriticChars.TryGetValue(c, out char o))
textBuilder.Append(o);
else
textBuilder.Append(c);
}
return text;
return textBuilder.ToString();
}
static Dictionary<string, string> diacriticChars = new Dictionary<string, string>
static readonly Dictionary<char, char> diacriticChars = new()
{
{ "ä", "a" },
{ "æǽ", "ae" },
{ "œ", "oe" },
{ "ö", "o" },
{ "ü", "u" },
{ "Ä", "A" },
{ "Ü", "U" },
{ "Ö", "O" },
{ "ÀÁÂÃÄÅǺĀĂĄǍΆẢẠẦẪẨẬẰẮẴẲẶ", "A" },
{ "àáâãåǻāăąǎảạầấẫẩậằắẵẳặа", "a" },
{ "ÇĆĈĊČ", "C" },
{ "çćĉċč", "c" },
{ "ÐĎĐ", "D" },
{ "ðďđ", "d" },
{ "ÈÉÊËĒĔĖĘĚΈẼẺẸỀẾỄỂỆ", "E" },
{ "èéêëēĕėęěẽẻẹềếễểệе", "e" },
{ "ĜĞĠĢ", "G" },
{ "ĝğġģ", "g" },
{ "ĤĦΉ", "H" },
{ "ĥħ", "h" },
{ "ÌÍÎÏĨĪĬǏĮİΊΪỈỊЇ", "I" },
{ "ìíîïĩīĭǐįıίϊỉịї", "i" },
{ "Ĵ", "J" },
{ "ĵ", "j" },
{ "Ķ", "K" },
{ "ķ", "k" },
{ "ĹĻĽĿŁ", "L" },
{ "ĺļľŀł", "l" },
{ "ÑŃŅŇ", "N" },
{ "ñńņňʼn", "n" },
{ "ÒÓÔÕŌŎǑŐƠØǾΌỎỌỒỐỖỔỘỜỚỠỞỢ", "O" },
{ "òóôõōŏǒőơøǿºόỏọồốỗổộờớỡởợ", "o" },
{ "ŔŖŘ", "R" },
{ "ŕŗř", "r" },
{ "ŚŜŞȘŠ", "S" },
{ "śŝşșš", "s" },
{ "ȚŢŤŦТ", "T" },
{ "țţťŧ", "t" },
{ "ÙÚÛŨŪŬŮŰŲƯǓǕǗǙǛŨỦỤỪỨỮỬỰ", "U" },
{ "ùúûũūŭůűųưǔǖǘǚǜủụừứữửự", "u" },
{ "ÝŸŶΎΫỲỸỶỴ", "Y" },
{ "Й", "Й" },
{ "й", "и" },
{ "ýÿŷỳỹỷỵ", "y" },
{ "Ŵ", "W" },
{ "ŵ", "w" },
{ "ŹŻŽ", "Z" },
{ "źżž", "z" },
{ "ÆǼ", "AE" },
{ "ß", "ss" },
{ "IJ", "IJ" },
{ "ij", "ij" },
{ "Œ", "OE" },
{ "Ё", "Е" },
{ "ё", "е" },
{ 'ä', 'a' }, { 'æ', 'a' }, { 'ǽ', 'a' }, { 'œ', 'o' }, { 'ö', 'o' }, { 'ü', 'u' },
{ 'Ä', 'A' }, { 'Ü', 'U' }, { 'Ö', 'O' }, { 'À', 'A' }, { 'Á', 'A' }, { 'Â', 'A' },
{ 'Ã', 'A' }, { 'Å', 'A' }, { 'Ǻ', 'A' }, { 'Ā', 'A' }, { 'Ă', 'A' }, { 'Ą', 'A' },
{ 'Ǎ', 'A' }, { 'Ά', 'A' }, { 'Ả', 'A' }, { 'Ạ', 'A' }, { 'Ầ', 'A' }, { 'Ấ', 'A' },
{ 'Ẫ', 'A' }, { 'Ẩ', 'A' }, { 'Ậ', 'A' }, { 'à', 'a' }, { 'á', 'a' }, { 'â', 'a' },
{ 'ã', 'a' }, { 'å', 'a' }, { 'ǻ', 'a' }, { 'ā', 'a' }, { 'ă', 'a' }, { 'ą', 'a' },
{ 'ǎ', 'a' }, { 'ả', 'a' }, { 'ạ', 'a' }, { 'Ç', 'C' }, { 'Ć', 'C' }, { 'Ĉ', 'C' },
{ 'Ċ', 'C' }, { 'Č', 'C' }, { 'ç', 'c' }, { 'ć', 'c' }, { 'ĉ', 'c' }, { 'ċ', 'c' },
{ 'č', 'c' }, { 'Ð', 'D' }, { 'Ď', 'D' }, { 'Đ', 'D' }, { 'ð', 'd' }, { 'ď', 'd' },
{ 'đ', 'd' }, { 'È', 'E' }, { 'É', 'E' }, { 'Ê', 'E' }, { 'Ë', 'E' }, { 'Ē', 'E' },
{ 'Ĕ', 'E' }, { 'Ė', 'E' }, { 'Ę', 'E' }, { 'Ě', 'E' }, { 'Έ', 'E' }, { 'Ẽ', 'E' },
{ 'Ẻ', 'E' }, { 'Ẹ', 'E' }, { 'Ề', 'E' }, { 'Ế', 'E' }, { 'Ễ', 'E' }, { 'Ể', 'E' },
{ 'Ệ', 'E' }, { 'è', 'e' }, { 'é', 'e' }, { 'ê', 'e' }, { 'ë', 'e' }, { 'ē', 'e' },
{ 'ĕ', 'e' }, { 'ė', 'e' }, { 'ę', 'e' }, { 'ě', 'e' }, { 'ẽ', 'e' }, { 'ẻ', 'e' },
{ 'ẹ', 'e' }, { 'Ĝ', 'G' }, { 'Ğ', 'G' }, { 'Ġ', 'G' }, { 'Ģ', 'G' }, { 'ĝ', 'g' },
{ 'ğ', 'g' }, { 'ġ', 'g' }, { 'ģ', 'g' }, { 'Ĥ', 'H' }, { 'Ħ', 'H' }, { 'Ή', 'H' },
{ 'ĥ', 'h' }, { 'ħ', 'h' }, { 'Ì', 'I' }, { 'Í', 'I' }, { 'Î', 'I' }, { 'Ï', 'I' },
{ 'Ĩ', 'I' }, { 'Ī', 'I' }, { 'Ĭ', 'I' }, { 'Ǐ', 'I' }, { 'Į', 'I' }, { 'İ', 'I' },
{ 'Ί', 'I' }, { 'Ϊ', 'I' }, { 'Ỉ', 'I' }, { 'Ị', 'I' }, { 'Ї', 'I' }, { 'ì', 'i' },
{ 'í', 'i' }, { 'î', 'i' }, { 'ï', 'i' }, { 'ĩ', 'i' }, { 'ī', 'i' }, { 'ĭ', 'i' },
{ 'ǐ', 'i' }, { 'į', 'i' }, { 'ı', 'i' }, { 'ΰ', 'y' }, { 'Ĵ', 'J' }, { 'ĵ', 'j' },
{ 'Ķ', 'K' }, { 'ķ', 'k' }, { 'Ĺ', 'L' }, { 'Ļ', 'L' }, { 'Ľ', 'L' }, { 'Ŀ', 'L' },
{ 'Ł', 'L' }, { 'ĺ', 'l' }, { 'ļ', 'l' }, { 'ľ', 'l' }, { 'ŀ', 'l' }, { 'ł', 'l' },
{ 'Ñ', 'N' }, { 'Ń', 'N' }, { 'Ņ', 'N' }, { 'Ň', 'N' }, { 'ñ', 'n' }, { 'ń', 'n' },
{ 'ņ', 'n' }, { 'ň', 'n' }, { 'ʼn', 'n' }, { 'Ò', 'O' }, { 'Ó', 'O' }, { 'Ô', 'O' },
{ 'Õ', 'O' }, { 'Ō', 'O' }, { 'Ŏ', 'O' }, { 'Ǒ', 'O' }, { 'Ő', 'O' }, { 'Ơ', 'O' },
{ 'Ø', 'O' }, { 'Ǿ', 'O' }, { 'Ό', 'O' }, { 'Ỏ', 'O' }, { 'Ọ', 'O' }, { 'Ồ', 'O' },
{ 'Ố', 'O' }, { 'Ỗ', 'O' }, { 'Ổ', 'O' }, { 'Ộ', 'O' }, { 'Ờ', 'O' }, { 'Ớ', 'O' },
{ 'Ỡ', 'O' }, { 'Ở', 'O' }, { 'Ợ', 'O' }, { 'ò', 'o' }, { 'ó', 'o' }, { 'ô', 'o' },
{ 'õ', 'o' }, { 'ō', 'o' }, { 'ŏ', 'o' }, { 'ǒ', 'o' }, { 'ő', 'o' }, { 'ơ', 'o' },
{ 'ø', 'o' }, { 'ǿ', 'o' }, { 'º', 'o' }, { 'ό', 'o' }, { 'ỏ', 'o' }, { 'ọ', 'o' },
{ 'ồ', 'o' }, { 'ố', 'o' }, { 'ỗ', 'o' }, { 'ổ', 'o' }, { 'ộ', 'o' }, { 'ờ', 'o' },
{ 'ớ', 'o' }, { 'ỡ', 'o' }, { 'ở', 'o' }, { 'ợ', 'o' }, { 'Ŕ', 'R' }, { 'Ŗ', 'R' },
{ 'Ř', 'R' }, { 'ŕ', 'r' }, { 'ŗ', 'r' }, { 'ř', 'r' }, { 'Ś', 'S' }, { 'Ŝ', 'S' },
{ 'Ş', 'S' }, { 'Ș', 'S' }, { 'Š', 'S' }, { 'ś', 's' }, { 'ŝ', 's' }, { 'ş', 's' },
{ 'ș', 's' }, { 'š', 's' }, { 'Ț', 'T' }, { 'Ţ', 'T' }, { 'Ť', 'T' }, { 'Ŧ', 'T' },
{ 'Т', 'T' }, { 'ț', 't' }, { 'ţ', 't' }, { 'ť', 't' }, { 'ŧ', 't' }, { 'Ù', 'U' },
{ 'Ú', 'U' }, { 'Û', 'U' }, { 'Ũ', 'U' }, { 'Ū', 'U' }, { 'Ŭ', 'U' }, { 'Ů', 'U' },
{ 'Ű', 'U' }, { 'Ų', 'U' }, { 'Ư', 'U' }, { 'Ǔ', 'U' }, { 'Ǖ', 'U' }, { 'Ǘ', 'U' },
{ 'Ǚ', 'U' }, { 'Ǜ', 'U' }, { 'Ủ', 'U' }, { 'Ụ', 'U' }, { 'Ừ', 'U' }, { 'ё', 'e' },
{ 'Ứ', 'U' }, { 'Ữ', 'U' }, { 'Ử', 'U' }, { 'Ự', 'U' }, { 'ù', 'u' }, { 'ú', 'u' },
{ 'û', 'u' }, { 'ũ', 'u' }, { 'ū', 'u' }, { 'ŭ', 'u' }, { 'ů', 'u' }, { 'ű', 'u' },
{ 'ų', 'u' }, { 'ư', 'u' }, { 'ǔ', 'u' }, { 'ǖ', 'u' }, { 'ǘ', 'u' }, { 'ǚ', 'u' },
{ 'ǜ', 'u' }, { 'ủ', 'u' }, { 'ụ', 'u' }, { 'ừ', 'u' }, { 'ứ', 'u' }, { 'ữ', 'u' },
{ 'ử', 'u' }, { 'ự', 'u' }, { 'Ý', 'Y' }, { 'Ÿ', 'Y' }, { 'Ŷ', 'Y' }, { 'Ύ', 'Y' },
{ 'Ϋ', 'Y' }, { 'Ỳ', 'Y' }, { 'Ỹ', 'Y' }, { 'Ỷ', 'Y' }, { 'Ỵ', 'Y' }, { 'й', 'и' },
{ 'ý', 'y' }, { 'ÿ', 'y' }, { 'ŷ', 'y' }, { 'ỳ', 'y' }, { 'ỹ', 'y' }, { 'ỷ', 'y' },
{ 'ỵ', 'y' }, { 'Ŵ', 'W' }, { 'ŵ', 'w' }, { 'Ź', 'Z' }, { 'Ż', 'Z' }, { 'Ž', 'Z' },
{ 'ź', 'z' }, { 'ż', 'z' }, { 'ž', 'z' }, { 'Æ', 'A' }, { 'ß', 's' }, { 'Œ', 'O' },
{ 'Ё', 'E' },
};
}
class RateLimitedSemaphore
{
private readonly int maxCount;
private readonly TimeSpan resetTimeSpan;
private readonly SemaphoreSlim semaphore;
private long nextResetTimeTicks;
private readonly object resetTimeLock = new object();
public RateLimitedSemaphore(int maxCount, TimeSpan resetTimeSpan)
{
this.maxCount = maxCount;
this.resetTimeSpan = resetTimeSpan;
this.semaphore = new SemaphoreSlim(maxCount, maxCount);
this.nextResetTimeTicks = (DateTimeOffset.UtcNow + this.resetTimeSpan).UtcTicks;
}
private void TryResetSemaphore()
{
if (!(DateTimeOffset.UtcNow.UtcTicks > Interlocked.Read(ref this.nextResetTimeTicks)))
return;
lock (this.resetTimeLock)
{
var currentTime = DateTimeOffset.UtcNow;
if (currentTime.UtcTicks > Interlocked.Read(ref this.nextResetTimeTicks))
{
int releaseCount = this.maxCount - this.semaphore.CurrentCount;
if (releaseCount > 0)
this.semaphore.Release(releaseCount);
var newResetTimeTicks = (currentTime + this.resetTimeSpan).UtcTicks;
Interlocked.Exchange(ref this.nextResetTimeTicks, newResetTimeTicks);
}
}
}
public async Task WaitAsync()
{
TryResetSemaphore();
var semaphoreTask = this.semaphore.WaitAsync();
while (!semaphoreTask.IsCompleted)
{
var ticks = Interlocked.Read(ref this.nextResetTimeTicks);
var nextResetTime = new DateTimeOffset(new DateTime(ticks, DateTimeKind.Utc));
var delayTime = nextResetTime - DateTimeOffset.UtcNow;
var delayTask = delayTime >= TimeSpan.Zero ? Task.Delay(delayTime) : Task.CompletedTask;
await Task.WhenAny(semaphoreTask, delayTask);
TryResetSemaphore();
}
}
}
public static class Logger
{
public static Verbosity verbosity { get; set; } = Verbosity.Normal;
public static void Log(string message, Verbosity messageLevel)
{
if (messageLevel <= verbosity)
{
Console.WriteLine(message);
}
}
public static void Error(string message)
{
Log(message, Verbosity.Error);
}
public static void Warning(string message)
{
Log(message, Verbosity.Warning);
}
public static void Info(string message)
{
Log(message, Verbosity.Normal);
}
public static void Verbose(string message)
{
Log(message, Verbosity.Verbose);
}
}

610
slsk-batchdl/YouTube.cs
View file

@ -1,610 +0,0 @@
using Google.Apis.YouTube.v3;
using Google.Apis.Services;
using System.Xml;
using YoutubeExplode;
using System.Text.RegularExpressions;
using YoutubeExplode.Common;
using System.Diagnostics;
using HtmlAgilityPack;
using System.Collections.Concurrent;
public static class YouTube
{
private static YoutubeClient? youtube = new YoutubeClient();
private static YouTubeService? youtubeService = null;
public static string apiKey = "";
public static async Task<(string, List<Track>)> GetTracksApi(string url, int max = int.MaxValue, int offset = 0)
{
StartService();
string playlistId = await UrlToId(url);
var playlistRequest = youtubeService.Playlists.List("snippet");
playlistRequest.Id = playlistId;
var playlistResponse = playlistRequest.Execute();
string playlistName = playlistResponse.Items[0].Snippet.Title;
var playlistItemsRequest = youtubeService.PlaylistItems.List("snippet,contentDetails");
playlistItemsRequest.PlaylistId = playlistId;
playlistItemsRequest.MaxResults = Math.Min(max, 100);
var tracksDict = await GetDictYtExplode(url, max, offset);
var tracks = new List<Track>();
int count = 0;
while (playlistItemsRequest != null && count < max + offset)
{
var playlistItemsResponse = playlistItemsRequest.Execute();
foreach (var playlistItem in playlistItemsResponse.Items)
{
if (count >= offset)
{
if (tracksDict.ContainsKey(playlistItem.Snippet.ResourceId.VideoId))
tracks.Add(tracksDict[playlistItem.Snippet.ResourceId.VideoId]);
else
{
var title = "";
var uploader = "";
var length = 0;
var desc = "";
var videoRequest = youtubeService.Videos.List("contentDetails,snippet");
videoRequest.Id = playlistItem.Snippet.ResourceId.VideoId;
var videoResponse = videoRequest.Execute();
title = playlistItem.Snippet.Title;
if (videoResponse.Items.Count == 0)
continue;
uploader = videoResponse.Items[0].Snippet.ChannelTitle;
length = (int)XmlConvert.ToTimeSpan(videoResponse.Items[0].ContentDetails.Duration).TotalSeconds;
desc = videoResponse.Items[0].Snippet.Description;
Track track = await ParseTrackInfo(title, uploader, playlistItem.Snippet.ResourceId.VideoId, length, desc);
tracks.Add(track);
}
}
if (++count >= max + offset)
break;
}
if (tracksDict.Count >= 200 && !Console.IsOutputRedirected)
{
Console.SetCursorPosition(0, Console.CursorTop);
Console.Write($"Loaded: {tracks.Count}");
}
playlistItemsRequest.PageToken = playlistItemsResponse.NextPageToken;
if (playlistItemsRequest.PageToken == null || count >= max + offset)
playlistItemsRequest = null;
else
playlistItemsRequest.MaxResults = Math.Min(offset + max - count, 100);
}
Console.WriteLine();
return (playlistName, tracks);
}
// requestInfoIfNeeded=true is way too slow
public static async Task<Track> ParseTrackInfo(string title, string uploader, string id, int length, string desc = "", bool requestInfoIfNeeded=false)
{
(string title, string uploader, int length, string desc) info = ("", "", -1, "");
var track = new Track();
track.URI = id;
uploader = uploader.Replace("–", "-").Trim().RemoveConsecutiveWs();
title = title.Replace("–", "-").Replace(" -- ", " - ").Trim().RemoveConsecutiveWs();
var artist = uploader;
var trackTitle = title;
if (artist.EndsWith(" - Topic"))
{
artist = artist.Substring(0, artist.Length - 7).Trim();
trackTitle = title;
if (artist == "Various Artists")
{
if (desc == "" && requestInfoIfNeeded && id != "")
{
info = await GetVideoInfo(id);
desc = info.desc;
}
if (desc != "")
{
var lines = desc.Split(new[] { "\r\n", "\r", "\n" }, StringSplitOptions.RemoveEmptyEntries);
var dotLine = lines.FirstOrDefault(line => line.Contains(" · "));
if (dotLine != null)
artist = dotLine.Split(new[] { " · " }, StringSplitOptions.None)[1];
}
}
}
else
{
track.ArtistMaybeWrong = !title.ContainsWithBoundary(artist, true) && !desc.ContainsWithBoundary(artist, true);
var split = title.Split(" - ", StringSplitOptions.TrimEntries | StringSplitOptions.RemoveEmptyEntries);
if (split.Length == 2)
{
artist = split[0];
trackTitle = split[1];
track.ArtistMaybeWrong = false;
}
else if (split.Length > 2)
{
int index = Array.FindIndex(split, s => s.ContainsWithBoundary(artist, true));
if (index != -1 && index < split.Length - 1)
{
artist = split[index];
trackTitle = String.Join(" - ", split[(index + 1)..]);
track.ArtistMaybeWrong = false;
}
}
if (track.ArtistMaybeWrong && requestInfoIfNeeded && desc == "")
{
info = await GetVideoInfo(id);
track.ArtistMaybeWrong = !info.desc.ContainsWithBoundary(artist, true);
}
}
if (length <= 0 && id != "" && requestInfoIfNeeded)
{
if (info.length > 0)
length = info.length;
else
{
info = await GetVideoInfo(id);
length = info.length;
}
}
track.Length = length;
track.Artist = artist;
track.Title = trackTitle;
return track;
}
public static async Task<(string title, string uploader, int length, string desc)> GetVideoInfo(string id)
{
(string title, string uploader, int length, string desc) o = ("", "", -1, "");
try
{
var vid = await youtube.Videos.GetAsync(id);
o.title = vid.Title;
o.uploader = vid.Author.ChannelTitle;
o.desc = vid.Description;
o.length = (int)vid.Duration.Value.TotalSeconds;
}
catch
{
if (apiKey != "")
{
try
{
StartService();
var videoRequest = youtubeService.Videos.List("contentDetails,snippet");
videoRequest.Id = id;
var videoResponse = videoRequest.Execute();
o.title = videoResponse.Items[0].Snippet.Title;
o.uploader = videoResponse.Items[0].Snippet.ChannelTitle;
o.length = (int)XmlConvert.ToTimeSpan(videoResponse.Items[0].ContentDetails.Duration).TotalSeconds;
o.desc = videoResponse.Items[0].Snippet.Description;
}
catch { }
}
}
return o;
}
public static void StartService()
{
if (youtubeService == null)
{
if (apiKey == "")
throw new Exception("No API key");
youtubeService = new YouTubeService(new BaseClientService.Initializer()
{
ApiKey = apiKey,
ApplicationName = "slsk-batchdl"
});
}
}
public static void StopService()
{
youtubeService = null;
}
public static async Task<Dictionary<string, Track>> GetDictYtExplode(string url, int max = int.MaxValue, int offset = 0)
{
var youtube = new YoutubeClient();
var playlist = await youtube.Playlists.GetAsync(url);
var tracks = new Dictionary<string, Track>();
int count = 0;
await foreach (var video in youtube.Playlists.GetVideosAsync(playlist.Id))
{
if (count >= offset && count < offset + max)
{
var title = video.Title;
var uploader = video.Author.Title;
var ytId = video.Id.Value;
var length = (int)video.Duration.Value.TotalSeconds;
var track = await ParseTrackInfo(title, uploader, ytId, length);
tracks[ytId] = track;
}
if (count++ >= offset + max)
break;
}
return tracks;
}
public static async Task<string> GetPlaylistTitle(string url)
{
var youtube = new YoutubeClient();
var playlist = await youtube.Playlists.GetAsync(url);
return playlist.Title;
}
public static async Task<(string, List<Track>)> GetTracksYtExplode(string url, int max = int.MaxValue, int offset = 0)
{
var youtube = new YoutubeClient();
var playlist = await youtube.Playlists.GetAsync(url);
var playlistTitle = playlist.Title;
var tracks = new List<Track>();
int count = 0;
await foreach (var video in youtube.Playlists.GetVideosAsync(playlist.Id))
{
if (count >= offset && count < offset + max)
{
var title = video.Title;
var uploader = video.Author.Title;
var ytId = video.Id.Value;
var length = (int)video.Duration.Value.TotalSeconds;
var track = await ParseTrackInfo(title, uploader, ytId, length);
tracks.Add(track);
}
if (count++ >= offset + max)
break;
}
return (playlistTitle, tracks);
}
public static async Task<string> UrlToId(string url)
{
var playlist = await youtube.Playlists.GetAsync(url);
return playlist.Id.ToString();
}
public class YouTubeArchiveRetriever
{
private HttpClient _client;
public YouTubeArchiveRetriever()
{
_client = new HttpClient();
_client.Timeout = TimeSpan.FromSeconds(10);
}
public async Task<List<Track>> RetrieveDeleted(string url, bool printFailed = true)
{
var deletedVideoUrls = new BlockingCollection<string>();
int totalCount = 0;
int archivedCount = 0;
var tracks = new ConcurrentBag<Track>();
var noArchive = new ConcurrentBag<string>();
var failRetrieve = new ConcurrentBag<string>();
int workerCount = 4;
var workers = new List<Task>();
var consoleLock = new object();
void updateInfo()
{
lock (consoleLock)
{
if (!Console.IsOutputRedirected)
{
string info = "Deleted metadata total/archived/retrieved: ";
Console.SetCursorPosition(0, Console.CursorTop);
Console.Write($"{info}{totalCount}/{archivedCount}/{tracks.Count}");
}
}
}
var process = new Process
{
StartInfo = new ProcessStartInfo
{
FileName = "yt-dlp",
Arguments = $"--ignore-no-formats-error --no-warn --match-filter \"!uploader\" --print webpage_url {url}",
RedirectStandardOutput = true,
UseShellExecute = false,
CreateNoWindow = true,
},
EnableRaisingEvents = true
};
process.OutputDataReceived += (sender, e) =>
{
if (!string.IsNullOrWhiteSpace(e.Data))
{
deletedVideoUrls.Add(e.Data);
Interlocked.Increment(ref totalCount);
updateInfo();
}
};
process.Exited += (sender, e) =>
{
deletedVideoUrls.CompleteAdding();
};
process.Start();
process.BeginOutputReadLine();
for (int i = 0; i < workerCount; i++)
{
workers.Add(Task.Run(async () =>
{
foreach (var videoUrl in deletedVideoUrls.GetConsumingEnumerable())
{
var waybackUrls = await GetOldestArchiveUrls(videoUrl, limit: 2);
if (waybackUrls != null && waybackUrls.Count > 0)
{
Interlocked.Increment(ref archivedCount);
bool good = false;
foreach (var waybackUrl in waybackUrls)
{
var (title, uploader, duration) = await GetVideoDetails(waybackUrl);
if (!string.IsNullOrWhiteSpace(title))
{
var track = await ParseTrackInfo(title, uploader, waybackUrl, duration);
track.Other = $"{{\"t\":\"{title.Trim()}\",\"u\":\"{uploader.Trim()}\"}}";
tracks.Add(track);
good = true;
break;
}
}
if (!good)
{
failRetrieve.Add(waybackUrls[0]);
}
}
else
{
noArchive.Add(videoUrl);
}
updateInfo();
}
}));
}
await Task.WhenAll(workers);
process.WaitForExit();
deletedVideoUrls.CompleteAdding();
Console.WriteLine();
if (printFailed)
{
if (archivedCount < totalCount)
{
Console.WriteLine("No archived version found for the following:");
foreach (var x in noArchive)
Console.WriteLine($" {x}");
Console.WriteLine();
}
if (tracks.Count < archivedCount)
{
Console.WriteLine("Failed to parse archived version for the following:");
foreach (var x in failRetrieve)
Console.WriteLine($" {x}");
Console.WriteLine();
}
}
return tracks.ToList();
}
private async Task<List<string>> GetOldestArchiveUrls(string url, int limit)
{
var url2 = $"http://web.archive.org/cdx/search/cdx?url={url}&fl=timestamp,original&filter=statuscode:200&sort=timestamp:asc&limit={limit}";
HttpResponseMessage response = null;
for (int i = 0; i < 3; i++)
{
try {
response = await _client.GetAsync(url2);
break;
}
catch (Exception e) { }
}
if (response == null) return null;
if (response.IsSuccessStatusCode)
{
var content = await response.Content.ReadAsStringAsync();
var lines = content.Split("\n").Where(line => !string.IsNullOrWhiteSpace(line)).ToList();
if (lines.Count > 0)
{
for (int i = 0; i < lines.Count; i++)
{
var parts = lines[i].Split(" ");
var timestamp = parts[0];
var originalUrl = parts[1];
lines[i] = $"http://web.archive.org/web/{timestamp}/{originalUrl}";
}
return lines;
}
}
return null;
}
public async Task<(string title, string uploader, int duration)> GetVideoDetails(string url)
{
var web = new HtmlWeb();
var doc = await web.LoadFromWebAsync(url);
var titlePatterns = new[]
{
"//h1[@id='video_title']",
"//meta[@name='title']",
};
var usernamePatterns = new[]
{
"//div[@id='userInfoDiv']/b/a",
"//a[contains(@class, 'contributor')]",
"//a[@id='watch-username']",
"//a[contains(@class, 'author')]",
"//div[@class='yt-user-info']/a",
"//div[@id='upload-info']//yt-formatted-string/a",
"//span[@itemprop='author']//link[@itemprop='name']",
"//a[contains(@class, 'yt-user-name')]",
};
string getItem(string[] patterns)
{
foreach (var pattern in patterns)
{
var node = doc.DocumentNode.SelectSingleNode(pattern);
if (node != null)
{
var res = "";
if (pattern.StartsWith("//meta") || pattern.Contains("@itemprop"))
res = node.GetAttributeValue("content", "");
else
res = node.InnerText;
if (!string.IsNullOrEmpty(res))
return Utils.UnHtmlString(res);
}
}
return "";
}
var title = getItem(titlePatterns);
if (string.IsNullOrEmpty(title))
{
var pattern = @"document\.title\s*=\s*""(.+?) - YouTube"";";
var match = Regex.Match(doc.Text, pattern);
if (match.Success)
title = match.Groups[1].Value;
}
var username = getItem(usernamePatterns);
int duration = -1;
var node = doc.DocumentNode.SelectSingleNode("//meta[@itemprop='duration']");
if (node != null)
{
try
{
duration = (int)XmlConvert.ToTimeSpan(node.GetAttributeValue("content", "")).TotalSeconds;
}
catch { }
}
return (title, username, duration);
}
}
public static async Task<List<(int length, string id, string title)>> YtdlpSearch(Track track)
{
Process process = new Process();
ProcessStartInfo startInfo = new ProcessStartInfo();
startInfo.FileName = "yt-dlp";
string search = track.Artist != "" ? $"{track.Artist} - {track.Title}" : track.Title;
startInfo.Arguments = $"\"ytsearch3:{search}\" --print \"%(duration>%s)s === %(id)s === %(title)s\"";
startInfo.RedirectStandardOutput = true;
startInfo.RedirectStandardError = true;
startInfo.UseShellExecute = false;
process.StartInfo = startInfo;
process.OutputDataReceived += (sender, e) => { Console.WriteLine(e.Data); };
process.ErrorDataReceived += (sender, e) => { Console.WriteLine(e.Data); };
process.Start();
List<(int, string, string)> results = new List<(int, string, string)>();
string output;
Regex regex = new Regex(@"^(\d+) === ([\w-]+) === (.+)$");
while ((output = process.StandardOutput.ReadLine()) != null)
{
Match match = regex.Match(output);
if (match.Success)
{
int seconds = int.Parse(match.Groups[1].Value);
string id = match.Groups[2].Value;
string title = match.Groups[3].Value;
results.Add((seconds, id, title));
}
}
process.WaitForExit();
return results;
}
public static async Task<string> YtdlpDownload(string id, string savePathNoExt, string ytdlpArgument="")
{
Process process = new Process();
ProcessStartInfo startInfo = new ProcessStartInfo();
if (ytdlpArgument == "")
ytdlpArgument = "\"{id}\" -f bestaudio/best -ci -o \"{savepath-noext}.%(ext)s\" -x";
startInfo.FileName = "yt-dlp";
startInfo.Arguments = ytdlpArgument
.Replace("{id}", id)
.Replace("{savepath}", savePathNoExt)
.Replace("{savepath-noext}", savePathNoExt)
.Replace("{savedir}", Path.GetDirectoryName(savePathNoExt));
startInfo.RedirectStandardOutput = true;
startInfo.RedirectStandardError = true;
startInfo.UseShellExecute = false;
process.StartInfo = startInfo;
//process.OutputDataReceived += (sender, e) => { Console.WriteLine(e.Data); };
//process.ErrorDataReceived += (sender, e) => { Console.WriteLine(e.Data); };
process.Start();
process.WaitForExit();
if (File.Exists(savePathNoExt + ".opus"))
return savePathNoExt + ".opus";
string parentDirectory = Path.GetDirectoryName(savePathNoExt);
string[] musicFiles = Directory.GetFiles(parentDirectory, "*", SearchOption.TopDirectoryOnly)
.Where(file => Utils.IsMusicFile(file))
.ToArray();
if (musicFiles.Length > 0)
return musicFiles[0];
return "";
}
}

20
slsk-batchdl/slsk-batchdl.csproj
View file

@ -1,22 +1,22 @@
<Project Sdk="Microsoft.NET.Sdk">
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net6.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<Nullable>enable</Nullable>
<AssemblyName>sldl</AssemblyName>
<VersionPrefix>2.3</VersionPrefix>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|AnyCPU'">
<DefineConstants>$(DefineConstants);TRACE</DefineConstants>
<DebugType>portable</DebugType>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|AnyCPU'" />
<ItemGroup>
<Compile Remove="Test.cs" />
</ItemGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|AnyCPU'">
<DebugType>portable</DebugType>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Goblinfactory.ProgressBar" Version="1.0.0" />
@ -27,7 +27,11 @@
<PackageReference Include="SpotifyAPI.Web" Version="7.1.1" />
<PackageReference Include="SpotifyAPI.Web.Auth" Version="7.1.1" />
<PackageReference Include="TagLibSharp" Version="2.3.0" />
<PackageReference Include="YoutubeExplode" Version="6.3.16" />
<PackageReference Include="YoutubeExplode" Version="6.4.0" />
</ItemGroup>
<ItemGroup>
<Compile Remove="Test.cs" Condition="'$(Configuration)' == 'Release'" />
</ItemGroup>
</Project>