mirror/slsk-batchdl
1
0
Fork 0
mirror of https://github.com/fiso64/slsk-batchdl.git synced 2024-12-22 14:32:40 +00:00
Code Issues Activity

commit

Browse source
This commit is contained in:
fiso64 2024-09-01 19:52:50 +02:00
parent d56d68356a
commit 7cd634e54f
20 changed files with 849 additions and 390 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

55
README.md
View file

@ -1,6 +1,6 @@
# slsk-batchdl
A batch downloader for Soulseek built with Soulseek.NET. Accepts CSV files as well as Spotify and YouTube urls.
An automatic downloader for Soulseek built with Soulseek.NET. Accepts CSV files as well as Spotify and YouTube urls.
See the [examples](#examples-1).
## Index
@ -41,7 +41,6 @@ Usage: sldl <input> [OPTIONS]
```
General Options
-p, --path <path> Download directory
-f, --folder <name> Subfolder name. Set to '.' to output directly to --path
--input-type <type> Force set input type, [csv|youtube|spotify|bandcamp|string]
--name-format <format> Name format for downloaded tracks. See --help name-format
@ -189,9 +188,6 @@ Usage: sldl <input> [OPTIONS]
-t, --interactive Interactive mode, allows to select the folder and images
--album-track-count <num> Specify the exact number of tracks in the album. Add a + or
- for inequalities, e.g '5+' for five or more tracks.
--album-ignore-fails Do not skip to the next source and do not delete all
successfully downloaded files if one of the files in the
folder fails to download
--album-art <option> Retrieve additional images after downloading the album:
'default': No additional images
'largest': Download from the folder with the largest image
@ -199,11 +195,17 @@ Usage: sldl <input> [OPTIONS]
--album-art-only Only download album art for the provided album
--no-browse-folder Do not automatically browse user shares to get all files in
in the folder
--failed-album-path Path to move all album files to when one of the items from
the directory fails to download. Set to 'delete' to delete
the files instead. Set to 'disable' keep it where it is.
Default: {configured output dir}/failed
```
```
Aggregate Download
-g, --aggregate Aggregate download mode: Find and download all distinct
songs associated with the provided artist, album, or title.
--aggregate-length-tol <tol> Max length tolerance in seconds to consider two tracks or
albums equal. (Default: 3)
--min-shares-aggregate <num> Minimum number of shares of a track or album for it to be
downloaded in aggregate mode. (Default: 2)
--relax-filtering Slightly relax file filtering in aggregate mode to include
@ -226,7 +228,7 @@ Usage: sldl <input> [OPTIONS]
## Input types
The input type is usually determined automatically. To force a specific input type, set
--input-type [spotify|youtube|csv|string|bandcamp]. The following input types are available:
--input-type [spotify|youtube|csv|string|bandcamp|list]. The following input types are available:
### CSV file
Path to a local CSV file: Use a csv file containing track info of the songs to download.
@ -290,6 +292,7 @@ artist
album
length (in seconds)
artist-maybe-wrong
album-track-count
```
Example inputs and their interpretations:
```
@ -302,6 +305,17 @@ Input String | Artist | Title | Album | Lengt
'artist=AR, title=T, album=AL' | AR | T | AL |
```
### List
A path to a text file where each line has the following form:
```
"some input" "conditions" "preferred conditions"
"album=Album" "format=mp3; br > 128" "br >= 320"
```
Where "some input" is any of the above input types. The quotes can be omitted if the field
contains no spaces. The conditions and preferred conditions fields are added on top of the
configured conditions and can also be omitted. List input must be manually activated with
--input-type=list.
## Download modes
### Normal
@ -313,24 +327,18 @@ Input String | Artist | Title | Album | Lengt
string or csv row has no track title, or when -a/--album is enabled.
### Aggregate
With -g/--aggregate, sldl will first perform an ordinary search for the input, then attempt to
group the results into distinct songs and download one of each kind. A common use case is
finding all remixes of a song or printing all songs by an artist that are not your music dir.
Two files are considered equal if their inferred track title and artist name are equal
(ignoring case and some special characters), and their lengths are within --length-tol of each
other.
Note that this mode is not 100% reliable, which is why --min-shares-aggregate is set to 2 by
default, i.e. any song that is shared only once will be ignored.
With -g/--aggregate, sldl performs an ordinary search for the input then attempts to
group the results into distinct songs and download one of each kind, starting with the one
which is shared by the most users.
Note that --min-shares-aggregate is 2 by default, which means that songs shared by only
one user will be ignored.
### Album Aggregate
Activated when --album and --aggregate are enabled, in this mode sldl searches for the query
and groups results into distinct albums. Two folders are considered same if they have the
same number of audio files, and the durations of the files are within --length-tol of each
other (or within 3 seconds if length-tol is not configured). If both folders have exactly one
audio file with similar lengths, also checks if the inferred title and artist name coincide.
More reliable than normal aggregate due to much simpler grouping logic.
Note that --min-shares-aggregate is 2 by default, which means that folders shared only once
will be ignored.
Activated when both --album and --aggregate are enabled. sldl will group shares and download
one of each distinct album, starting with the one shared by the most users. It's
recommended to pair this with --interactive.
Note that --min-shares-aggregate is 2 by default, which means that albums shared by only
one user will be ignored.
## Searching
@ -407,7 +415,7 @@ only satisfies the format condition. Run with --print "results-full" to reveal t
ranking with this option due to the bitrate and samplerate checks.
Conditions can also be supplied as a semicolon-delimited string with --cond and --pref, e.g
--cond "br>=320;f=mp3,ogg;sr<96000"
--cond "br >= 320; format = mp3,ogg; sr < 96000".
## Name format
@ -443,7 +451,6 @@ track Track number
disc Disc number
filename Soulseek filename without extension
foldername Soulseek folder name
default-foldername Default sldl folder name
extractor Name of the extractor used (CSV/Spotify/YouTube/etc)
```

149
slsk-batchdl/Config.cs
View file

@ -21,13 +21,10 @@ static class Config
AcceptNoLength = false,
};
public static string parentFolder = Directory.GetCurrentDirectory();
public static string parentDir = Directory.GetCurrentDirectory();
public static string input = "";
public static string outputFolder = "";
public static string m3uFilePath = "";
public static string musicDir = "";
public static string folderName = "";
public static string defaultFolderName = "";
public static string spotifyId = "";
public static string spotifySecret = "";
public static string spotifyToken = "";
@ -49,11 +46,11 @@ static class Config
public static string onComplete = "";
public static string confPath = "";
public static string profile = "";
public static string failedAlbumPath = "";
public static bool aggregate = false;
public static bool album = false;
public static bool albumArtOnly = false;
public static bool interactiveMode = false;
public static bool albumIgnoreFails = false;
public static bool setAlbumMinTrackCount = true;
public static bool setAlbumMaxTrackCount = false;
public static bool skipNotFound = false;
@ -94,6 +91,7 @@ static class Config
public static int listenPort = 49998;
public static int searchesPerTime = 34;
public static int searchRenewTime = 220;
public static int aggregateLengthTol = 3;
public static double fastSearchMinUpSpeed = 1.0;
public static Track regexToReplace = new();
public static Track regexReplaceBy = new();
@ -105,19 +103,22 @@ static class Config
public static SkipMode skipModeMusicDir = SkipMode.Name;
public static PrintOption printOption = PrintOption.None;
static readonly Dictionary<string, (List<string> args, string? cond)> profiles = new();
static readonly HashSet<string> appliedProfiles = new();
static bool hasConfiguredM3uMode = false;
static bool confPathChanged = false;
static string[] arguments;
public static bool HasAutoProfiles { get; private set; } = false;
public static bool DoNotDownload => (printOption & (PrintOption.Results | PrintOption.Tracks)) != 0;
public static bool PrintTracks => (printOption & PrintOption.Tracks) != 0;
public static bool PrintResults => (printOption & PrintOption.Results) != 0;
public static bool PrintTracksFull => (printOption & PrintOption.Tracks) != 0 && (printOption & PrintOption.Full) != 0;
public static bool PrintResultsFull => (printOption & PrintOption.Results) != 0 && (printOption & PrintOption.Full) != 0;
public static bool DeleteAlbumOnFail => failedAlbumPath == "delete";
public static bool IgnoreAlbumFail => failedAlbumPath == "disable";
static readonly Dictionary<string, (List<string> args, string? cond)> profiles = new();
static readonly HashSet<string> appliedProfiles = new();
static bool hasConfiguredM3uMode = false;
static bool confPathChanged = false;
static string[] arguments;
static FileConditions? prevConds = null;
static FileConditions? prevPrefConds = null;
public static bool ParseArgsAndReadConfig(string[] args)
{
@ -218,7 +219,7 @@ static class Config
}
public static void PostProcessArgs() // must be run after Program.trackLists has been assigned
public static void PostProcessArgs()
{
if (DoNotDownload || debugInfo)
concurrentProcesses = 1;
@ -227,38 +228,21 @@ static class Config
if (DoNotDownload)
m3uOption = M3uOption.None;
else if (!hasConfiguredM3uMode)
{
if (inputType == InputType.String)
else if (!hasConfiguredM3uMode && inputType == InputType.String)
m3uOption = M3uOption.None;
else if (!aggregate && !(skipExisting && (skipMode == SkipMode.M3u || skipMode == SkipMode.M3uCond))
&& Program.trackLists != null && !Program.trackLists.Flattened(true, false, true).Skip(1).Any())
{
m3uOption = M3uOption.None;
}
}
if (albumArtOnly && albumArtOption == AlbumArtOption.Default)
albumArtOption = AlbumArtOption.Largest;
parentFolder = Utils.ExpandUser(parentFolder);
m3uFilePath = Utils.ExpandUser(m3uFilePath);
musicDir = Utils.ExpandUser(musicDir);
if (folderName.Length == 0)
folderName = defaultFolderName;
if (folderName == ".")
folderName = "";
folderName = folderName.Replace('\\', '/');
folderName = string.Join('/', folderName.Split('/').Select(x => x.ReplaceInvalidChars(invalidReplaceStr).Trim()));
folderName = folderName.Replace('/', Path.DirectorySeparatorChar);
outputFolder = Path.Join(parentFolder, folderName);
nameFormat = nameFormat.Trim();
if (m3uFilePath.Length == 0)
m3uFilePath = Path.Join(outputFolder, (folderName.Length == 0 ? "playlist" : folderName) + ".m3u8");
parentDir = Utils.ExpandUser(parentDir);
m3uFilePath = Utils.ExpandUser(m3uFilePath);
musicDir = Utils.ExpandUser(musicDir);
failedAlbumPath = Utils.ExpandUser(failedAlbumPath);
if (failedAlbumPath.Length == 0)
failedAlbumPath = Path.Join(parentDir, "failed");
}
@ -313,7 +297,7 @@ static class Config
}
public static void UpdateArgs(TrackListEntry tle)
public static void UpdateProfiles(TrackListEntry tle)
{
if (DoNotDownload)
return;
@ -491,9 +475,32 @@ static class Config
}
static void ParseConditions(FileConditions cond, string input)
public static void AddTemporaryConditions(FileConditionsPatch? cond, FileConditionsPatch? prefCond)
{
static void UpdateMinMax(string value, string condition, ref int min, ref int max)
if (cond != null)
{
prevConds = necessaryCond;
necessaryCond = necessaryCond.With(cond);
}
if (prefCond != null)
{
prevPrefConds = preferredCond;
preferredCond = preferredCond.With(prefCond);
}
}
public static void RestoreConditions()
{
if (prevConds != null)
necessaryCond = prevConds;
if (prevPrefConds != null)
preferredCond = prevPrefConds;
}
public static FileConditionsPatch ParseConditions(string input)
{
static void UpdateMinMax(string value, string condition, ref int? min, ref int? max)
{
if (condition.Contains(">="))
min = int.Parse(value);
@ -507,6 +514,8 @@ static class Config
min = max = int.Parse(value);
}
var cond = new FileConditionsPatch();
var tr = StringSplitOptions.TrimEntries | StringSplitOptions.RemoveEmptyEntries;
string[] conditions = input.Split(';', tr);
foreach (string condition in conditions)
@ -569,6 +578,8 @@ static class Config
throw new ArgumentException($"Unknown condition '{condition}'");
}
}
return cond;
}
@ -614,21 +625,18 @@ static class Config
"spotify" => InputType.Spotify,
"bandcamp" => InputType.Bandcamp,
"string" => InputType.String,
"list" => InputType.List,
_ => throw new ArgumentException($"Invalid input type '{args[i]}'"),
};
break;
case "-p":
case "--path":
parentFolder = args[++i];
parentDir = args[++i];
break;
case "-c":
case "--config":
confPath = args[++i];
break;
case "-f":
case "--folder":
folderName = args[++i];
break;
case "-m":
case "--md":
case "--music-dir":
@ -942,9 +950,9 @@ static class Config
preferredCond = new FileConditions();
necessaryCond = new FileConditions();
break;
case "--aif":
case "--album-ignore-fails":
setFlag(ref albumIgnoreFails, ref i);
case "--fap":
case "--failed-album-path":
failedAlbumPath = args[++i];
break;
case "-t":
case "--interactive":
@ -1063,12 +1071,12 @@ static class Config
case "--c":
case "--cond":
case "--conditions":
ParseConditions(necessaryCond, args[++i]);
necessaryCond.With(ParseConditions(args[++i]));
break;
case "--pc":
case "--pref":
case "--preferred-conditions":
ParseConditions(preferredCond, args[++i]);
preferredCond.With(ParseConditions(args[++i]));
break;
case "--nmsc":
case "--no-modify-share-count":
@ -1177,6 +1185,10 @@ static class Config
case "--skip-existing-pref-cond":
setFlag(ref skipExistingPrefCond, ref i);
break;
case "--alt":
case "--aggregate-length-tol":
aggregateLengthTol = int.Parse(args[++i]);
break;
default:
throw new ArgumentException($"Unknown argument: {args[i]}");
}
@ -1193,4 +1205,41 @@ static class Config
}
}
}
public static string[] GetArgsArray(string commandLine)
{
var args = new List<string>();
var currentArg = new StringBuilder();
bool inQuotes = false;
for (int i = 0; i < commandLine.Length; i++)
{
char c = commandLine[i];
if (c == '\"')
{
inQuotes = !inQuotes;
}
else if (c == ' ' && !inQuotes)
{
if (currentArg.Length > 0)
{
args.Add(currentArg.ToString());
currentArg.Clear();
}
}
else
{
currentArg.Append(c);
}
}
if (currentArg.Length > 0)
{
args.Add(currentArg.ToString());
}
return args.ToArray();
}
}

70
slsk-batchdl/Data.cs
View file

@ -100,38 +100,51 @@ namespace Data
public class TrackListEntry
{
public List<List<Track>> list;
public List<List<Track>>? list;
public Track source;
public bool needSourceSearch = false;
public bool sourceCanBeSkipped = false;
public bool needSkipExistingAfterSearch = false;
public bool gotoNextAfterSearch = false;
public bool placeInSubdir = false;
public string? defaultFolderName = null;
public FileConditionsPatch? additionalConds = null;
public FileConditionsPatch? additionalPrefConds = null;
public TrackListEntry()
public TrackListEntry(TrackType trackType)
{
list = new List<List<Track>>();
source = new Track();
this.source = new Track() { Type = trackType };
SetDefaults();
}
public TrackListEntry(Track source)
{
list = new List<List<Track>>();
this.source = source;
needSourceSearch = source.Type != TrackType.Normal;
needSkipExistingAfterSearch = source.Type == TrackType.Aggregate;
gotoNextAfterSearch = source.Type == TrackType.AlbumAggregate;
sourceCanBeSkipped = source.Type != TrackType.Normal
&& source.Type != TrackType.Aggregate
&& source.Type != TrackType.AlbumAggregate;
SetDefaults();
}
public TrackListEntry(List<List<Track>> list, Track source)
{
this.list = list;
this.source = source;
SetDefaults();
}
public TrackListEntry(List<List<Track>> list, Track source, bool needSourceSearch = false, bool sourceCanBeSkipped = false,
bool needSkipExistingAfterSearch = false, bool gotoNextAfterSearch = false, string? defaultFoldername = null)
{
this.list = list;
this.source = source;
this.needSourceSearch = needSourceSearch;
this.sourceCanBeSkipped = sourceCanBeSkipped;
this.needSkipExistingAfterSearch = needSkipExistingAfterSearch;
this.gotoNextAfterSearch = gotoNextAfterSearch;
this.defaultFolderName = defaultFoldername;
}
public void SetDefaults()
{
needSourceSearch = source.Type != TrackType.Normal;
needSkipExistingAfterSearch = source.Type == TrackType.Aggregate;
gotoNextAfterSearch = source.Type == TrackType.AlbumAggregate;
@ -140,16 +153,14 @@ namespace Data
&& source.Type != TrackType.AlbumAggregate;
}
public TrackListEntry(List<List<Track>> list, Track source, bool needSearch, bool placeInSubdir,
bool sourceCanBeSkipped, bool needSkipExistingAfterSearch, bool gotoNextAfterSearch)
public void AddTrack(Track track)
{
this.list = list;
this.source = source;
this.needSourceSearch = needSearch;
this.placeInSubdir = placeInSubdir;
this.sourceCanBeSkipped = sourceCanBeSkipped;
this.needSkipExistingAfterSearch = needSkipExistingAfterSearch;
this.gotoNextAfterSearch = gotoNextAfterSearch;
if (list == null)
list = new List<List<Track>>() { new List<Track>() { track } };
else if (list.Count == 0)
list.Add(new List<Track>() { track });
else
list[0].Add(track);
}
}
@ -174,7 +185,7 @@ namespace Data
}
else
{
res.AddEntry(new TrackListEntry());
res.AddEntry(new TrackListEntry(TrackType.Normal));
res.AddTrackToLast(track);
bool hasNext;
@ -252,11 +263,13 @@ namespace Data
if (tle.source.Type == TrackType.Album && aggregate)
{
tle.source.Type = TrackType.AlbumAggregate;
tle.SetDefaults();
newLists.Add(tle);
}
else if (tle.source.Type == TrackType.Aggregate && album)
{
tle.source.Type = TrackType.AlbumAggregate;
tle.SetDefaults();
newLists.Add(tle);
}
else if (tle.source.Type == TrackType.Normal && (album || aggregate))
@ -270,7 +283,9 @@ namespace Data
else if (aggregate)
track.Type = TrackType.Aggregate;
newLists.Add(new TrackListEntry(track));
var newTle = new TrackListEntry(track);
newTle.defaultFolderName = tle.defaultFolderName;
newLists.Add(newTle);
}
}
else
@ -284,16 +299,11 @@ namespace Data
public void SetListEntryOptions()
{
// place downloads in subdirs if there is more than one special (album/aggregate) download
bool placeInSubdirs = Flattened(true, false, true).Skip(1).Any();
if (placeInSubdirs)
{
// aggregate downloads will be placed in subfolders by default
foreach (var tle in lists)
{
if (tle.source.Type != TrackType.Normal)
tle.placeInSubdir = true;
}
if (tle.source.Type == TrackType.Aggregate || tle.source.Type == TrackType.AlbumAggregate)
tle.defaultFolderName = Path.Join(tle.defaultFolderName, tle.source.ToString(true));
}
}

16
slsk-batchdl/Download.cs
View file

@ -17,7 +17,7 @@ using SlDictionary = System.Collections.Concurrent.ConcurrentDictionary<string,
static class Download
{
public static async Task DownloadFile(SearchResponse response, Soulseek.File file, string filePath, Track track, ProgressBar progress, CancellationTokenSource? searchCts = null)
public static async Task DownloadFile(SearchResponse response, Soulseek.File file, string filePath, Track track, ProgressBar progress, CancellationTokenSource cts, CancellationTokenSource? searchCts = null)
{
if (Config.DoNotDownload)
throw new Exception();
@ -42,16 +42,10 @@ static class Download
try
{
using var cts = new CancellationTokenSource();
using var outputStream = new FileStream(filePath, FileMode.Create);
var wrapper = new DownloadWrapper(origPath, response, file, track, cts, progress);
downloads.TryAdd(file.Filename, wrapper);
// Attempt to make it resume downloads after a network interruption.
// Does not work: The resumed download will be queued until it goes stale.
// The host (slskd) reports that "Another upload to {user} is already in progress"
// when attempting to resume. Must wait until timeout, which can take minutes.
int maxRetries = 3;
int retryCount = 0;
while (true)
@ -65,10 +59,12 @@ static class Download
break;
}
catch (SoulseekClientException)
catch (SoulseekClientException e)
{
retryCount++;
Printing.WriteLine($"Error while downloading: {e}", ConsoleColor.DarkYellow, debugOnly: true);
if (retryCount >= maxRetries || IsConnectedAndLoggedIn())
throw;
@ -86,13 +82,13 @@ static class Download
throw;
}
try { searchCts?.Cancel(); }
catch { }
try { searchCts?.Cancel(); } catch { }
try { Utils.Move(filePath, origPath); }
catch (IOException) { Printing.WriteLine($"Failed to rename .incomplete file", ConsoleColor.DarkYellow, true); }
downloads.TryRemove(file.Filename, out var x);
if (x != null)
{
lock (x)

8
slsk-batchdl/Enums.cs
View file

@ -37,6 +37,7 @@ namespace Enums
Spotify,
Bandcamp,
String,
List,
None,
}
@ -85,4 +86,11 @@ namespace Enums
Normal,
Verbose
}
public enum AlbumFailOption
{
Ignore,
Keep,
Delete,
}
}

25
slsk-batchdl/Extractors/Bandcamp.cs
View file

@ -16,17 +16,17 @@ namespace Extractors
return input.IsInternetUrl() && input.Contains("bandcamp.com");
}
public async Task<TrackLists> GetTracks(int maxTracks, int offset, bool reverse)
public async Task<TrackLists> GetTracks(string input, int maxTracks, int offset, bool reverse)
{
var trackLists = new TrackLists();
bool isTrack = Config.input.Contains("/track/");
bool isAlbum = !isTrack && Config.input.Contains("/album/");
bool isTrack = input.Contains("/track/");
bool isAlbum = !isTrack && input.Contains("/album/");
bool isArtist =!isTrack && !isAlbum;
if (isArtist)
{
Console.WriteLine("Retrieving bandcamp artist discography..");
string artistUrl = Config.input.TrimEnd('/');
string artistUrl = input.TrimEnd('/');
if (!artistUrl.EndsWith("/music"))
artistUrl += "/music";
@ -51,17 +51,14 @@ namespace Extractors
foreach (var item in root.GetProperty("discography").EnumerateArray())
{
//ItemType = item.GetProperty("item_type").GetString(),
var t = new Track()
var track = new Track()
{
Album = item.GetProperty("title").GetString(),
Artist = item.GetProperty("artist_name").GetString() ?? item.GetProperty("band_name").GetString(),
Type = TrackType.Album,
};
var tle = new TrackListEntry()
{
source = t,
placeInSubdir = true,
};
var tle = new TrackListEntry(track);
tle.defaultFolderName = track.Artist;
trackLists.AddEntry(tle);
}
}
@ -69,7 +66,7 @@ namespace Extractors
{
Console.WriteLine("Retrieving bandcamp item..");
var web = new HtmlWeb();
var doc = await web.LoadFromWebAsync(Config.input);
var doc = await web.LoadFromWebAsync(input);
var nameSection = doc.DocumentNode.SelectSingleNode("//div[@id='name-section']");
var name = nameSection.SelectSingleNode(".//h2[contains(@class, 'trackTitle')]").InnerText.UnHtmlString().Trim();
@ -91,8 +88,6 @@ namespace Extractors
if (Config.setAlbumMaxTrackCount)
track.MaxAlbumTrackCount = n;
}
Config.defaultFolderName = track.ToString(true).ReplaceInvalidChars(Config.invalidReplaceStr).Trim();
}
else
{
@ -101,10 +96,8 @@ namespace Extractors
//var timeParts = doc.DocumentNode.SelectSingleNode("//span[@class='time_total']").InnerText.Trim().Split(':');
var track = new Track() { Artist = artist, Title = name, Album = album };
trackLists.AddEntry(new());
trackLists.AddEntry(new TrackListEntry(TrackType.Normal));
trackLists.AddTrackToLast(track);
Config.defaultFolderName = ".";
}
}

28
slsk-batchdl/Extractors/Csv.cs
View file

@ -6,7 +6,8 @@ namespace Extractors
{
public class CsvExtractor : IExtractor
{
object csvLock = new();
string? csvFilePath = null;
readonly object csvLock = new();
int csvColumnCount = -1;
public static bool InputMatches(string input)
@ -15,28 +16,26 @@ namespace Extractors
return !input.IsInternetUrl() && input.EndsWith(".csv");
}
public async Task<TrackLists> GetTracks(int maxTracks, int offset, bool reverse)
public async Task<TrackLists> GetTracks(string input, int maxTracks, int offset, bool reverse)
{
if (!File.Exists(Config.input))
if (!File.Exists(input))
throw new FileNotFoundException("CSV file not found");
var tracks = await ParseCsvIntoTrackInfo(Config.input, Config.artistCol, Config.trackCol, Config.lengthCol,
csvFilePath = input;
var tracks = await ParseCsvIntoTrackInfo(input, Config.artistCol, Config.trackCol, Config.lengthCol,
Config.albumCol, Config.descCol, Config.ytIdCol, Config.trackCountCol, Config.timeUnit, Config.ytParse);
if (reverse)
tracks.Reverse();
var trackLists = TrackLists.FromFlattened(tracks.Skip(offset).Take(maxTracks));
var csvName = Path.GetFileNameWithoutExtension(input);
foreach (var tle in trackLists.lists)
{
if (tle.source.Type != TrackType.Normal)
{
tle.placeInSubdir = true;
tle.defaultFolderName = csvName;
}
}
Config.defaultFolderName = Path.GetFileNameWithoutExtension(Config.input);
return trackLists;
}
@ -45,16 +44,16 @@ namespace Extractors
{
lock (csvLock)
{
if (File.Exists(Config.input))
if (File.Exists(csvFilePath))
{
try
{
string[] lines = File.ReadAllLines(Config.input, System.Text.Encoding.UTF8);
string[] lines = File.ReadAllLines(csvFilePath, System.Text.Encoding.UTF8);
if (track.CsvRow > -1 && track.CsvRow < lines.Length)
{
lines[track.CsvRow] = new string(',', Math.Max(0, csvColumnCount - 1));
Utils.WriteAllLines(Config.input, lines, '\n');
Utils.WriteAllLines(csvFilePath, lines, '\n');
}
}
catch (Exception e)
@ -193,7 +192,7 @@ namespace Extractors
return tracks;
}
double ParseTrackLength(string duration, string format)
static double ParseTrackLength(string duration, string format)
{
if (string.IsNullOrEmpty(format))
throw new ArgumentException("Duration format string empty");
@ -224,6 +223,5 @@ namespace Extractors
return totalSeconds;
}
}
}

146
slsk-batchdl/Extractors/List.cs Normal file
View file

@ -0,0 +1,146 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Data;
using Enums;
namespace Extractors
{
public class ListExtractor : IExtractor
{
string? listFilePath = null;
readonly object fileLock = new object();
public static bool InputMatches(string input)
{
return !input.IsInternetUrl();
}
public async Task<TrackLists> GetTracks(string input, int maxTracks, int offset, bool reverse)
{
if (!File.Exists(input))
throw new FileNotFoundException("List file not found");
listFilePath = input;
var lines = File.ReadAllLines(input);
var trackLists = new TrackLists();
int step = reverse ? -1 : 1;
int start = reverse ? lines.Length - 1 : 0;
int count = 0;
int added = 0;
string foldername = Path.GetFileNameWithoutExtension(input);
for (int i = start; i < lines.Length && i >= 0; i += step)
{
var line = lines[i].Trim();
if (line.Length == 0 || line.StartsWith('#')) continue;
if (count++ < offset)
continue;
if (added >= maxTracks)
break;
var fields = ParseLine(line);
var (_, ex) = ExtractorRegistry.GetMatchingExtractor(fields[0]);
var tl = await ex.GetTracks(fields[0], int.MaxValue, 0, false);
foreach (var tle in tl.lists)
{
if (fields.Count >= 2)
tle.additionalConds = Config.ParseConditions(fields[1]);
if (fields.Count >= 3)
tle.additionalPrefConds = Config.ParseConditions(fields[2]);
tle.defaultFolderName = foldername;
}
if (tl.lists.Count == 1)
tl[0].source.CsvRow = i;
trackLists.lists.AddRange(tl.lists);
added++;
}
return trackLists;
}
static List<string> ParseLine(string input)
{
var fields = new List<string>();
bool inQuotes = false;
var currentField = new StringBuilder();
input = input.Replace('\t', ' ');
for (int i = 0; i < input.Length; i++)
{
char c = input[i];
if (c == '"')
{
inQuotes = !inQuotes;
}
else if (c == ' ' && !inQuotes)
{
if (currentField.Length > 0)
{
fields.Add(currentField.ToString());
currentField.Clear();
}
while (i < input.Length - 1 && input[i + 1] == ' ')
{
i++;
}
}
else
{
currentField.Append(c);
}
}
if (currentField.Length > 0)
{
fields.Add(currentField.ToString());
}
return fields;
}
public async Task RemoveTrackFromSource(Track track)
{
lock (fileLock)
{
if (File.Exists(listFilePath))
{
try
{
string[] lines = File.ReadAllLines(listFilePath, Encoding.UTF8);
if (track.CsvRow > -1 && track.CsvRow < lines.Length)
{
lines[track.CsvRow] = "";
Utils.WriteAllLines(listFilePath, lines, '\n');
}
}
catch (Exception e)
{
Printing.WriteLine($"Error removing from source: {e}", debugOnly: true);
}
}
}
}
}
}

34
slsk-batchdl/Extractors/Spotify.cs
View file

@ -19,37 +19,35 @@ namespace Extractors
return input == "spotify-likes" || input.IsInternetUrl() && input.Contains("spotify.com");
}
public async Task<TrackLists> GetTracks(int maxTracks, int offset, bool reverse)
public async Task<TrackLists> GetTracks(string input, int maxTracks, int offset, bool reverse)
{
var trackLists = new TrackLists();
int max = reverse ? int.MaxValue : maxTracks;
int off = reverse ? 0 : offset;
string playlistName = "";
bool needLogin = Config.input == "spotify-likes" || Config.removeTracksFromSource;
var tle = new TrackListEntry();
bool needLogin = input == "spotify-likes" || Config.removeTracksFromSource;
var tle = new TrackListEntry(TrackType.Normal);
if (needLogin && Config.spotifyToken.Length == 0 && (Config.spotifyId.Length == 0 || Config.spotifySecret.Length == 0))
{
Console.WriteLine("Error: Credentials are required when downloading liked music or removing from source playlists.");
Environment.Exit(0);
Environment.Exit(1);
}
spotifyClient = new Spotify(Config.spotifyId, Config.spotifySecret, Config.spotifyToken, Config.spotifyRefresh);
await spotifyClient.Authorize(needLogin, Config.removeTracksFromSource);
if (Config.input == "spotify-likes")
if (input == "spotify-likes")
{
Console.WriteLine("Loading Spotify likes..");
var tracks = await spotifyClient.GetLikes(max, off);
playlistName = "Spotify Likes";
tle.defaultFolderName = "Spotify Likes";
tle.list.Add(tracks);
}
else if (Config.input.Contains("/album/"))
else if (input.Contains("/album/"))
{
Console.WriteLine("Loading Spotify album..");
(var source, var tracks) = await spotifyClient.GetAlbum(Config.input);
playlistName = source.ToString(noInfo: true);
(var source, var tracks) = await spotifyClient.GetAlbum(input);
tle.source = source;
if (Config.setAlbumMinTrackCount)
@ -58,11 +56,11 @@ namespace Extractors
if (Config.setAlbumMaxTrackCount)
source.MaxAlbumTrackCount = tracks.Count;
}
else if (Config.input.Contains("/artist/"))
else if (input.Contains("/artist/"))
{
Console.WriteLine("Loading spotify artist..");
Console.WriteLine("Error: Spotify artist download currently not supported.");
Environment.Exit(0);
Environment.Exit(1);
}
else
{
@ -71,19 +69,21 @@ namespace Extractors
try
{
Console.WriteLine("Loading Spotify playlist");
(playlistName, playlistUri, tracks) = await spotifyClient.GetPlaylist(Config.input, max, off);
(var playlistName, playlistUri, tracks) = await spotifyClient.GetPlaylist(input, max, off);
tle.defaultFolderName = playlistName;
}
catch (SpotifyAPI.Web.APIException)
{
if (!needLogin && !spotifyClient.UsedDefaultCredentials)
{
await spotifyClient.Authorize(true, Config.removeTracksFromSource);
(playlistName, playlistUri, tracks) = await spotifyClient.GetPlaylist(Config.input, max, off);
(var playlistName, playlistUri, tracks) = await spotifyClient.GetPlaylist(input, max, off);
tle.defaultFolderName = playlistName;
}
else if (!needLogin)
{
Console.WriteLine("Spotify playlist not found (it may be set to private, but no credentials have been provided).");
Environment.Exit(0);
Console.WriteLine("Error: Spotify playlist not found (it may be set to private, but no credentials have been provided).");
Environment.Exit(1);
}
else throw;
}
@ -91,8 +91,6 @@ namespace Extractors
tle.list.Add(tracks);
}
Config.defaultFolderName = playlistName.ReplaceInvalidChars(Config.invalidReplaceStr);
trackLists.AddEntry(tle);
if (reverse)

40
slsk-batchdl/Extractors/String.cs
View file

@ -11,35 +11,33 @@ namespace Extractors
return !input.IsInternetUrl();
}
public async Task<TrackLists> GetTracks(int maxTracks, int offset, bool reverse)
public async Task<TrackLists> GetTracks(string input, int maxTracks, int offset, bool reverse)
{
var trackLists = new TrackLists();
var music = ParseTrackArg(Config.input, Config.album);
var music = ParseTrackArg(input, Config.album);
TrackListEntry tle;
if (Config.album || (music.Title.Length == 0 && music.Album.Length > 0))
{
music.Type = TrackType.Album;
trackLists.AddEntry(new TrackListEntry(music));
tle = new TrackListEntry(music);
}
else
{
trackLists.AddEntry(new TrackListEntry());
trackLists.AddTrackToLast(music);
tle = new TrackListEntry(TrackType.Normal);
tle.AddTrack(music);
}
if (Config.aggregate || Config.album || music.Type != TrackType.Normal)
Config.defaultFolderName = music.ToString(true).ReplaceInvalidChars(Config.invalidReplaceStr).Trim();
else
Config.defaultFolderName = ".";
trackLists.AddEntry(tle);
return trackLists;
}
static public Track ParseTrackArg(string input, bool isAlbum)
public static Track ParseTrackArg(string input, bool isAlbum)
{
input = input.Trim();
var track = new Track();
var keys = new string[] { "title", "artist", "length", "album", "artist-maybe-wrong" };
var keys = new string[] { "title", "artist", "length", "album", "artist-maybe-wrong", "album-track-count" };
void setProperty(string key, string value)
{
@ -60,6 +58,26 @@ namespace Extractors
case "artist-maybe-wrong":
if (value == "true") track.ArtistMaybeWrong = true;
break;
case "album-track-count":
if (value == "-1")
{
track.MinAlbumTrackCount = -1;
track.MaxAlbumTrackCount = -1;
}
else if (value.Last() == '-')
{
track.MaxAlbumTrackCount = int.Parse(value[..^1]);
}
else if (value.Last() == '+')
{
track.MinAlbumTrackCount = int.Parse(value[..^1]);
}
else
{
track.MinAlbumTrackCount = int.Parse(value);
track.MaxAlbumTrackCount = track.MinAlbumTrackCount;
}
break;
}
}

20
slsk-batchdl/Extractors/YouTube.cs
View file

@ -9,6 +9,7 @@ using HtmlAgilityPack;
using System.Collections.Concurrent;
using Data;
using Enums;
namespace Extractors
{
@ -20,7 +21,7 @@ namespace Extractors
return input.IsInternetUrl() && (input.Contains("youtu.be") || input.Contains("youtube.com"));
}
public async Task<TrackLists> GetTracks(int maxTracks, int offset, bool reverse)
public async Task<TrackLists> GetTracks(string input, int maxTracks, int offset, bool reverse)
{
var trackLists = new TrackLists();
int max = reverse ? int.MaxValue : maxTracks;
@ -35,24 +36,24 @@ namespace Extractors
{
Console.WriteLine("Getting deleted videos..");
var archive = new YouTube.YouTubeArchiveRetriever();
deleted = await archive.RetrieveDeleted(Config.input, printFailed: Config.deletedOnly);
deleted = await archive.RetrieveDeleted(input, printFailed: Config.deletedOnly);
}
if (!Config.deletedOnly)
{
if (YouTube.apiKey.Length > 0)
{
Console.WriteLine("Loading YouTube playlist (API)");
(name, tracks) = await YouTube.GetTracksApi(Config.input, max, off);
(name, tracks) = await YouTube.GetTracksApi(input, max, off);
}
else
{
Console.WriteLine("Loading YouTube playlist");
(name, tracks) = await YouTube.GetTracksYtExplode(Config.input, max, off);
(name, tracks) = await YouTube.GetTracksYtExplode(input, max, off);
}
}
else
{
name = await YouTube.GetPlaylistTitle(Config.input);
name = await YouTube.GetPlaylistTitle(input);
}
if (deleted != null)
{
@ -61,11 +62,12 @@ namespace Extractors
YouTube.StopService();
var tle = new TrackListEntry();
tle.list.Add(tracks);
trackLists.AddEntry(tle);
var tle = new TrackListEntry(TrackType.Normal);
Config.defaultFolderName = name.ReplaceInvalidChars(Config.invalidReplaceStr);
tle.defaultFolderName = name;
tle.list.Add(tracks);
trackLists.AddEntry(tle);
if (reverse)
{

21
slsk-batchdl/Extractors/_Extractor.cs
View file

@ -6,7 +6,7 @@ namespace Extractors
{
public interface IExtractor
{
Task<TrackLists> GetTracks(int maxTracks, int offset, bool reverse);
Task<TrackLists> GetTracks(string input, int maxTracks, int offset, bool reverse);
Task RemoveTrackFromSource(Track track) => Task.CompletedTask;
}
@ -19,18 +19,29 @@ namespace Extractors
(InputType.Spotify, SpotifyExtractor.InputMatches, () => new SpotifyExtractor()),
(InputType.Bandcamp, BandcampExtractor.InputMatches, () => new BandcampExtractor()),
(InputType.String, StringExtractor.InputMatches, () => new StringExtractor()),
(InputType.List, ListExtractor.InputMatches, () => new ListExtractor()),
};
public static (InputType, IExtractor?) GetMatchingExtractor(string input)
public static (InputType, IExtractor) GetMatchingExtractor(string input, InputType inputType = InputType.None)
{
foreach ((var inputType, var inputMatches, var extractor) in extractors)
if (string.IsNullOrEmpty(input))
throw new ArgumentException("Input string can not be null or empty.");
if (inputType != InputType.None)
{
var (t, _, e) = extractors.First(x => x.Item1 == inputType);
return (t, e());
}
foreach ((var type, var inputMatches, var extractor) in extractors)
{
if (inputMatches(input))
{
return (inputType, extractor());
return (type, extractor());
}
}
return (InputType.None, null);
throw new ArgumentException($"No matching extractor for input '{input}'");
}
}
}

73
slsk-batchdl/FileConditions.cs
View file

@ -41,6 +41,58 @@ public class FileConditions
BannedUsers = other.BannedUsers.ToArray();
}
public FileConditions With(FileConditionsPatch patch)
{
var cond = new FileConditions(this);
if (patch.LengthTolerance != null)
cond.LengthTolerance = patch.LengthTolerance.Value;
if (patch.MinBitrate != null)
cond.MinBitrate = patch.MinBitrate.Value;
if (patch.MaxBitrate != null)
cond.MaxBitrate = patch.MaxBitrate.Value;
if (patch.MinSampleRate != null)
cond.MinSampleRate = patch.MinSampleRate.Value;
if (patch.MaxSampleRate != null)
cond.MaxSampleRate = patch.MaxSampleRate.Value;
if (patch.MinBitDepth != null)
cond.MinBitDepth = patch.MinBitDepth.Value;
if (patch.MaxBitDepth != null)
cond.MaxBitDepth = patch.MaxBitDepth.Value;
if (patch.StrictTitle != null)
cond.StrictTitle = patch.StrictTitle.Value;
if (patch.StrictArtist != null)
cond.StrictArtist = patch.StrictArtist.Value;
if (patch.StrictAlbum != null)
cond.StrictAlbum = patch.StrictAlbum.Value;
if (patch.Formats != null)
cond.Formats = patch.Formats;
if (patch.BannedUsers != null)
cond.BannedUsers = patch.BannedUsers;
if (patch.StrictStringDiacrRemove != null)
cond.StrictStringDiacrRemove = patch.StrictStringDiacrRemove.Value;
if (patch.AcceptNoLength != null)
cond.AcceptNoLength = patch.AcceptNoLength.Value;
if (patch.AcceptMissingProps != null)
cond.AcceptMissingProps = patch.AcceptMissingProps.Value;
return cond;
}
public override bool Equals(object obj)
{
if (obj is FileConditions other)
@ -244,3 +296,24 @@ public class FileConditions
return "Satisfied";
}
}
public class FileConditionsPatch
{
public int? LengthTolerance = null;
public int? MinBitrate = null;
public int? MaxBitrate = null;
public int? MinSampleRate = null;
public int? MaxSampleRate = null;
public int? MinBitDepth = null;
public int? MaxBitDepth = null;
public bool? StrictTitle = null;
public bool? StrictArtist = null;
public bool? StrictAlbum = null;
public string[]? Formats = null;
public string[]? BannedUsers = null;
public bool? StrictStringDiacrRemove = null;
public bool? AcceptNoLength = null;
public bool? AcceptMissingProps = null;
}

78
slsk-batchdl/FileManager.cs
View file

@ -7,16 +7,15 @@ using System.Text.RegularExpressions;
using Data;
using Enums;
using System.ComponentModel;
public class FileManager
{
readonly TrackListEntry? tle;
readonly TrackListEntry tle;
readonly HashSet<Track> organized = new();
string? remoteCommonDir;
public FileManager() { }
public FileManager(TrackListEntry tle)
{
this.tle = tle;
@ -24,18 +23,24 @@ public class FileManager
public string GetSavePath(string sourceFname)
{
return $"{GetSavePathNoExt(sourceFname)}{Path.GetExtension(sourceFname)}";
return GetSavePathNoExt(sourceFname) + Path.GetExtension(sourceFname);
}
public string GetSavePathNoExt(string sourceFname)
{
string parent = Config.parentDir;
string name = Utils.GetFileNameWithoutExtSlsk(sourceFname);
name = name.ReplaceInvalidChars(Config.invalidReplaceStr);
string parent = Config.outputFolder;
if (tle != null && tle.placeInSubdir && remoteCommonDir != null)
if (tle.defaultFolderName != null)
{
parent = Path.Join(parent, tle.defaultFolderName.ReplaceInvalidChars(Config.invalidReplaceStr, removeSlash: false));
}
if (tle.source.Type == TrackType.Album)
{
if (remoteCommonDir == null)
throw new NullReferenceException("Remote common dir needs to be configured to organize album files");
string dirname = Path.GetFileName(remoteCommonDir);
string relpath = Path.GetRelativePath(remoteCommonDir, Utils.NormalizedPath(sourceFname));
parent = Path.Join(parent, dirname, Path.GetDirectoryName(relpath));
@ -51,17 +56,12 @@ public class FileManager
public void OrganizeAlbum(List<Track> tracks, List<Track>? additionalImages, bool remainingOnly = true)
{
if (tle == null)
throw new NullReferenceException("TrackListEntry should not be null.");
string outputFolder = Config.outputFolder;
foreach (var track in tracks.Where(t => !t.IsNotAudio))
{
if (remainingOnly && organized.Contains(track))
continue;
OrganizeAudio(tle, track, track.FirstDownload);
OrganizeAudio(track, track.FirstDownload);
}
bool onlyAdditionalImages = Config.nameFormat.Length == 0;
@ -83,20 +83,19 @@ public class FileManager
}
}
public void OrganizeAudio(TrackListEntry tle, Track track, Soulseek.File? file)
public void OrganizeAudio(Track track, Soulseek.File? file)
{
if (track.DownloadPath.Length == 0 || !Utils.IsMusicFile(track.DownloadPath))
{
return;
}
else if (Config.nameFormat.Length == 0)
if (Config.nameFormat.Length == 0)
{
organized.Add(track);
return;
}
string pathPart = SubstituteValues(Config.nameFormat, track, file);
string newFilePath = Path.Join(Config.outputFolder, pathPart + Path.GetExtension(track.DownloadPath));
string newFilePath = Path.Join(Config.parentDir, pathPart + Path.GetExtension(track.DownloadPath));
try
{
@ -141,15 +140,7 @@ public class FileManager
{
Directory.CreateDirectory(Path.GetDirectoryName(newPath));
Utils.Move(oldPath, newPath);
string x = Utils.NormalizedPath(Path.GetFullPath(Config.outputFolder));
string y = Utils.NormalizedPath(Path.GetDirectoryName(oldPath));
while (x.Length > 0 && y.StartsWith(x + '/') && Utils.FileCountRecursive(y) == 0)
{
Directory.Delete(y, true); // hopefully this is fine
y = Utils.NormalizedPath(Path.GetDirectoryName(y));
}
Utils.DeleteAncestorsIfEmpty(Path.GetDirectoryName(oldPath), Config.parentDir);
}
}
@ -172,25 +163,33 @@ public class FileManager
inner = inner[1..^1];
var options = inner.Split('|');
string chosenOpt = "";
string? chosenOpt = null;
foreach (var opt in options)
{
string[] parts = Regex.Split(opt, @"\([^\)]*\)");
string[] result = parts.Where(part => !string.IsNullOrWhiteSpace(part)).ToArray();
if (result.All(x => GetVarValue(x, file, slfile, track).Length > 0))
if (result.All(x => TryGetVarValue(x, file, slfile, track, out string res) && res.Length > 0))
{
chosenOpt = opt;
break;
}
}
if (chosenOpt == null)
{
chosenOpt = options[^1];
}
chosenOpt = Regex.Replace(chosenOpt, @"\([^()]*\)|[^()]+", match =>
{
if (match.Value.StartsWith("(") && match.Value.EndsWith(")"))
return match.Value[1..^1].ReplaceInvalidChars(Config.invalidReplaceStr, removeSlash: false);
else
return GetVarValue(match.Value, file, slfile, track);
{
TryGetVarValue(match.Value, file, slfile, track, out string res);
return res;
}
});
string old = match.Groups[1].Value;
@ -213,10 +212,8 @@ public class FileManager
return format;
}
string GetVarValue(string x, TagLib.File? file, Soulseek.File? slfile, Track track)
bool TryGetVarValue(string x, TagLib.File? file, Soulseek.File? slfile, Track track, out string res)
{
string res;
switch (x)
{
case "artist":
@ -249,23 +246,24 @@ public class FileManager
case "foldername":
if (remoteCommonDir == null || slfile == null)
{
return Utils.GetBaseNameSlsk(Utils.GetDirectoryNameSlsk(slfile?.Filename ?? ""));
res = Utils.GetBaseNameSlsk(Utils.GetDirectoryNameSlsk(slfile?.Filename ?? ""));
return true;
}
else
{
string d = Path.GetDirectoryName(Utils.NormalizedPath(slfile.Filename));
string r = Path.GetFileName(remoteCommonDir);
return Path.Join(r, Path.GetRelativePath(remoteCommonDir, d));
res = Path.Join(r, Path.GetRelativePath(remoteCommonDir, d));
return true;
}
case "default-foldername":
res = Config.defaultFolderName; break;
case "extractor":
res = Config.inputType.ToString(); break;
default:
res = ""; break;
res = x; return false;
}
return res.ReplaceInvalidChars(Config.invalidReplaceStr);
res = res.ReplaceInvalidChars(Config.invalidReplaceStr);
return true;
}
}

55
slsk-batchdl/Help.cs
View file

@ -19,8 +19,7 @@ public static class Help
General Options
-p, --path <path> Download directory
-f, --folder <name> Subfolder name. Set to '.' to output directly to --path
--input-type <type> Force set input type, [csv|youtube|spotify|bandcamp|string]
--input-type <type> [csv|youtube|spotify|bandcamp|string|list]
--name-format <format> Name format for downloaded tracks. See --help name-format
-n, --number <maxtracks> Download the first n tracks of a playlist
@ -161,9 +160,6 @@ public static class Help
-t, --interactive Interactive mode, allows to select the folder and images
--album-track-count <num> Specify the exact number of tracks in the album. Add a + or
- for inequalities, e.g '5+' for five or more tracks.
--album-ignore-fails Do not skip to the next source and do not delete all
successfully downloaded files if one of the files in the
folder fails to download
--album-art <option> Retrieve additional images after downloading the album:
'default': No additional images
'largest': Download from the folder with the largest image
@ -171,10 +167,16 @@ public static class Help
--album-art-only Only download album art for the provided album
--no-browse-folder Do not automatically browse user shares to get all files in
in the folder
--failed-album-path Path to move all album files to when one of the items from
the directory fails to download. Set to 'delete' to delete
the files instead. Set to the empty string """" to disable.
Default: {configured output dir}/failed
Aggregate Download
-g, --aggregate Aggregate download mode: Find and download all distinct
songs associated with the provided artist, album, or title.
--aggregate-length-tol <tol> Max length tolerance in seconds to consider two tracks or
albums equal. (Default: 3)
--min-shares-aggregate <num> Minimum number of shares of a track or album for it to be
downloaded in aggregate mode. (Default: 2)
--relax-filtering Slightly relax file filtering in aggregate mode to include
@ -196,7 +198,7 @@ public static class Help
Input types
The input type is usually determined automatically. To force a specific input type, set
--input-type [spotify|youtube|csv|string|bandcamp]. The following input types are available:
--input-type [spotify|youtube|csv|string|bandcamp|list]. The following input types are available:
CSV file
Path to a local CSV file: Use a csv file containing track info of the songs to download.
@ -259,6 +261,7 @@ public static class Help
album
length (in seconds)
artist-maybe-wrong
album-track-count
Example inputs and their interpretations:
Input String | Artist | Title | Album | Length
@ -268,6 +271,17 @@ public static class Help
'Foo - Bar' (with --album enabled) | Foo | | Bar |
'Artist - Title, length=42' | Artist | Title | | 42
'artist=AR, title=T, album=AL' | AR | T | AL |
List
A path to a text file where each line has the following form:
""some input"" ""conditions"" ""preferred conditions""
""album=Album"" ""format=mp3; br > 128"" ""br >= 320""
Where ""some input"" is any of the above input types. The quotes can be omitted if the field
contains no spaces. The conditions and preferred conditions fields are added on top of the
configured conditions and can also be omitted. List input must be manually activated with
--input-type=list.
";
const string downloadModesHelp = @"
@ -282,24 +296,18 @@ public static class Help
or csv row has no track title, or when -a/--album is enabled.
Aggregate
With -g/--aggregate, sldl will first perform an ordinary search for the input, then attempt to
group the results into distinct songs and download one of each kind. A common use case is
finding all remixes of a song or printing all songs by an artist that are not your music dir.
Two files are considered equal if their inferred track title and artist name are equal
(ignoring case and some special characters), and their lengths are within --length-tol of each
other.
Note that this mode is not 100% reliable, which is why --min-shares-aggregate is set to 2 by
default, i.e. any song that is shared only once will be ignored.
With -g/--aggregate, sldl performs an ordinary search for the input then attempts to
group the results into distinct songs and download one of each kind, starting with the one
which is shared by the most users.
Note that --min-shares-aggregate is 2 by default, which means that songs shared by only
one user will be ignored.
Album Aggregate
Activated when --album and --aggregate are enabled, in this mode sldl searches for the query
and groups results into distinct albums. Two folders are considered same if they have the
same number of audio files, and the durations of the files are within --length-tol of each
other (or within 3 seconds if length-tol is not configured). If both folders have exactly one
audio file with similar lengths, also checks if the inferred title and artist name coincide.
More reliable than normal aggregate due to much simpler grouping logic.
Note that --min-shares-aggregate is 2 by default, which means that folders shared only once
will be ignored.
Activated when both --album and --aggregate are enabled. sldl will group shares and download
one of each distinct album, starting with the one shared by the most users. It's
recommended to pair this with --interactive.
Note that --min-shares-aggregate is 2 by default, which means that albums shared by only
one user will be ignored.
";
const string searchHelp = @"
@ -378,7 +386,7 @@ public static class Help
ranking with this option due to the bitrate and samplerate checks.
Conditions can also be supplied as a semicolon-delimited string with --cond and --pref, e.g
--cond ""br>=320;f=mp3,ogg;sr<96000""
--cond ""br >= 320; format = mp3,ogg; sr < 96000"".
";
const string nameFormatHelp = @"
@ -414,7 +422,6 @@ public static class Help
disc Disc number
filename Soulseek filename without extension
foldername Soulseek folder name
default-foldername Default sldl folder name
extractor Name of the extractor used (CSV/Spotify/YouTube/etc)
";

33
slsk-batchdl/M3uEditor.cs
View file

@ -5,25 +5,34 @@ using System.Text;
public class M3uEditor
{
public string path { get; private set; }
string parent;
List<string> lines;
bool needFirstUpdate = false;
readonly TrackLists trackLists;
readonly string path;
readonly string parent;
readonly int offset = 0;
readonly M3uOption option = M3uOption.Index;
readonly Dictionary<string, Track> previousRunData = new(); // { track.ToKey(), track }
public M3uEditor(string m3uPath, TrackLists trackLists, M3uOption option, int offset = 0)
public M3uEditor(TrackLists trackLists, M3uOption option)
{
this.trackLists = trackLists;
this.offset = offset;
this.option = option;
this.path = Path.GetFullPath(m3uPath);
this.parent = Utils.NormalizedPath(Path.GetDirectoryName(path));
this.lines = ReadAllLines().ToList();
this.needFirstUpdate = option == M3uOption.All;
}
public M3uEditor(string path, TrackLists trackLists, M3uOption option) : this(trackLists, option)
{
SetPathAndLoad(path);
}
public void SetPathAndLoad(string path)
{
if (this.path == path)
return;
this.path = Path.GetFullPath(path);
parent = Utils.NormalizedPath(Path.GetDirectoryName(this.path));
lines = ReadAllLines().ToList();
LoadPreviousResults();
}
@ -72,7 +81,7 @@ public class M3uEditor
if (field == 0)
{
if (x.StartsWith("./"))
x = Path.Join(parent, x[2..]);
x = System.IO.Path.Join(parent, x[2..]);
track.DownloadPath = x;
}
else if (field == 1)
@ -116,7 +125,7 @@ public class M3uEditor
lock (trackLists)
{
bool needUpdate = false;
int index = 1 + offset;
int index = 1;
bool updateLine(string newLine)
{
@ -246,7 +255,7 @@ public class M3uEditor
{
string p = val.DownloadPath;
if (Utils.NormalizedPath(p).StartsWith(parent))
p = "./" + Path.GetRelativePath(parent, p); // prepend ./ for LoadPreviousResults to recognize that a rel. path is used
p = "./" + System.IO.Path.GetRelativePath(parent, p); // prepend ./ for LoadPreviousResults to recognize that a rel. path is used
var items = new string[]
{
@ -278,7 +287,7 @@ public class M3uEditor
if (track.DownloadPath.Length > 0)
{
if (track.DownloadPath.StartsWith(parent))
if (Utils.NormalizedPath(track.DownloadPath).StartsWith(parent))
return Path.GetRelativePath(parent, track.DownloadPath);
else
return track.DownloadPath;

283
slsk-batchdl/Program.cs
View file

@ -5,10 +5,12 @@ using System.Collections.Concurrent;
using System.Data;
using System.Diagnostics;
using System.Text.RegularExpressions;
using System.Net.Sockets;
using Data;
using Enums;
using FileSkippers;
using Extractors;
using static Printing;
using Directory = System.IO.Directory;
@ -54,14 +56,11 @@ static partial class Program
if (Config.input.Length == 0)
throw new ArgumentException($"No input provided");
(Config.inputType, extractor) = Extractors.ExtractorRegistry.GetMatchingExtractor(Config.input);
if (Config.inputType == InputType.None)
throw new ArgumentException($"No matching extractor for input '{Config.input}'");
(Config.inputType, extractor) = ExtractorRegistry.GetMatchingExtractor(Config.input, Config.inputType);
WriteLine($"Using extractor: {Config.inputType}", debugOnly: true);
trackLists = await extractor.GetTracks(Config.maxTracks, Config.offset, Config.reverse);
trackLists = await extractor.GetTracks(Config.input, Config.maxTracks, Config.offset, Config.reverse);
WriteLine("Got tracks", debugOnly: true);
@ -71,7 +70,7 @@ static partial class Program
Config.PostProcessArgs();
m3uEditor = new M3uEditor(Config.m3uFilePath, trackLists, Config.m3uOption, Config.offset);
m3uEditor = new M3uEditor(trackLists, Config.m3uOption);
InitFileSkippers();
@ -88,9 +87,25 @@ static partial class Program
bool needLogin = !Config.PrintTracks;
if (needLogin)
{
client = new SoulseekClient(new SoulseekClientOptions(listenPort: Config.listenPort));
var connectionOptions = new ConnectionOptions(configureSocket: (socket) =>
{
socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, true);
socket.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveRetryCount, 3);
socket.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveTime, 15);
socket.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveInterval, 15);
});
var clientOptions = new SoulseekClientOptions(
transferConnectionOptions: connectionOptions,
serverConnectionOptions: connectionOptions,
listenPort: Config.listenPort
);
client = new SoulseekClient(clientOptions);
if (!Config.useRandomLogin && (string.IsNullOrEmpty(Config.username) || string.IsNullOrEmpty(Config.password)))
throw new ArgumentException("No soulseek username or password");
await Login(Config.useRandomLogin);
Search.searchSemaphore = new RateLimitedSemaphore(Config.searchesPerTime, TimeSpan.FromSeconds(Config.searchRenewTime));
@ -113,8 +128,8 @@ static partial class Program
{
var cond = Config.skipExistingPrefCond ? Config.preferredCond : Config.necessaryCond;
if (Config.musicDir.Length == 0 || !Config.outputFolder.StartsWith(Config.musicDir, StringComparison.OrdinalIgnoreCase))
outputDirSkipper = FileSkipperRegistry.GetChecker(Config.skipMode, Config.outputFolder, cond, m3uEditor);
if (Config.musicDir.Length == 0 || !Config.parentDir.StartsWith(Config.musicDir, StringComparison.OrdinalIgnoreCase))
outputDirSkipper = FileSkipperRegistry.GetChecker(Config.skipMode, Config.parentDir, cond, m3uEditor);
if (Config.musicDir.Length > 0)
{
@ -171,6 +186,27 @@ static partial class Program
}
static void PrepareListEntry(TrackListEntry tle)
{
Config.RestoreConditions();
Config.UpdateProfiles(tle);
Config.AddTemporaryConditions(tle.additionalConds, tle.additionalPrefConds);
string m3uPath;
if (Config.m3uFilePath.Length > 0)
m3uPath = Config.m3uFilePath;
else
m3uPath = Path.Join(Config.parentDir, tle.defaultFolderName, "sldl.m3u");
m3uEditor.SetPathAndLoad(m3uPath);
PreprocessTracks(tle);
}
static async Task MainLoop()
{
for (int i = 0; i < trackLists.lists.Count; i++)
@ -179,13 +215,10 @@ static partial class Program
var tle = trackLists[i];
Config.UpdateArgs(tle);
PreprocessTracks(tle);
PrepareListEntry(tle);
var existing = new List<Track>();
var notFound = new List<Track>();
var responseData = new ResponseData();
if (Config.skipNotFound && !Config.PrintResults)
{
@ -247,13 +280,18 @@ static partial class Program
Console.WriteLine($"{tle.source.Type} download: {tle.source.ToString(true)}, searching..");
bool foundSomething = false;
var responseData = new ResponseData();
if (tle.source.Type == TrackType.Album)
{
tle.list = await Search.GetAlbumDownloads(tle.source, responseData);
foundSomething = tle.list.Count > 0;
}
else if (tle.source.Type == TrackType.Aggregate)
{
tle.list.Insert(0, await Search.GetAggregateTracks(tle.source, responseData));
foundSomething = tle.list.Count > 0;
}
else if (tle.source.Type == TrackType.AlbumAggregate)
{
@ -262,8 +300,27 @@ static partial class Program
foreach (var item in res)
{
var newSource = new Track(tle.source) { Type = TrackType.Album };
trackLists.AddEntry(new TrackListEntry(item, newSource, false, true, true, false, false));
var albumTle = new TrackListEntry(item, newSource, needSourceSearch: false, sourceCanBeSkipped: true);
albumTle.defaultFolderName = tle.defaultFolderName;
trackLists.AddEntry(albumTle);
}
foundSomething = res.Count > 0;
}
if (!foundSomething)
{
var lockedFiles = responseData.lockedFilesCount > 0 ? $" (Found {responseData.lockedFilesCount} locked files)" : "";
Console.WriteLine($"No results.{lockedFiles}");
if (!Config.PrintResults)
{
tle.source.State = TrackState.Failed;
tle.source.FailureReason = FailureReason.NoSuitableFileFound;
m3uEditor.Update();
}
continue;
}
if (Config.skipExisting && tle.needSkipExistingAfterSearch)
@ -284,18 +341,6 @@ static partial class Program
continue;
}
if (tle.needSourceSearch && (tle.list.Count == 0 || !tle.list.Any(x => x.Count > 0)))
{
string lockedFilesStr = responseData.lockedFilesCount > 0 ? $" (Found {responseData.lockedFilesCount} locked files)" : "";
Console.WriteLine($"No results.{lockedFilesStr}");
tle.source.State = TrackState.Failed;
tle.source.FailureReason = FailureReason.NoSuitableFileFound;
m3uEditor.Update();
continue;
}
m3uEditor.Update();
if (tle.source.Type != TrackType.Album)
@ -416,11 +461,15 @@ static partial class Program
var downloadTasks = tracks.Select(async (track, index) =>
{
await DownloadTask(tle, track, semaphore, organizer, null, false, true);
using var cts = new CancellationTokenSource();
await DownloadTask(tle, track, semaphore, organizer, cts, false, true, true);
m3uEditor.Update();
});
await Task.WhenAll(downloadTasks);
if (Config.removeTracksFromSource && tracks.All(t => t.State == TrackState.Downloaded || t.State == TrackState.AlreadyExists))
await extractor.RemoveTrackFromSource(tle.source);
}
@ -449,13 +498,6 @@ static partial class Program
organizer.SetRemoteCommonDir(soulseekDir);
if (!Config.noBrowseFolder && !Config.interactiveMode && !retrievedFolders.Contains(soulseekDir))
{
Console.WriteLine("Getting all files in folder...");
await Search.CompleteFolder(tracks, tracks[0].FirstResponse, soulseekDir);
retrievedFolders.Add(tracks[0].FirstUsername + '\\' + soulseekDir);
}
if (!Config.interactiveMode && !wasInteractive)
{
Console.WriteLine();
@ -463,36 +505,80 @@ static partial class Program
}
var semaphore = new SemaphoreSlim(Config.concurrentProcesses);
var cts = new CancellationTokenSource();
using var cts = new CancellationTokenSource();
try
{
var downloadTasks = tracks.Select(async track =>
await RunAlbumDownloads(tle, organizer, tracks, semaphore, cts);
if (!Config.noBrowseFolder && !retrievedFolders.Contains(soulseekDir))
{
await DownloadTask(tle, track, semaphore, organizer, cts, cancelOnFail: !Config.albumIgnoreFails, true);
});
await Task.WhenAll(downloadTasks);
Console.WriteLine("Getting all files in folder...");
int newFilesFound = await Search.CompleteFolder(tracks, tracks[0].FirstResponse, soulseekDir);
retrievedFolders.Add(tracks[0].FirstUsername + '\\' + soulseekDir);
if (newFilesFound > 0)
{
Console.WriteLine($"Found {newFilesFound} more files in the directory, downloading:");
await RunAlbumDownloads(tle, organizer, tracks, semaphore, cts);
}
else
{
Console.WriteLine("No more files found.");
}
}
succeeded = true;
break;
}
catch (OperationCanceledException) when (!Config.albumIgnoreFails)
catch (OperationCanceledException)
{
foreach (var track in tracks)
{
if (track.State == TrackState.Downloaded && File.Exists(track.DownloadPath))
{
try { File.Delete(track.DownloadPath); } catch { }
}
}
OnAlbumFail(tracks);
}
organizer.SetRemoteCommonDir(null);
tle.list.RemoveAt(index);
}
if (tracks != null && succeeded)
if (succeeded)
{
await OnAlbumSuccess(tle, tracks);
}
List<Track>? additionalImages = null;
if (Config.albumArtOnly || succeeded && Config.albumArtOption != AlbumArtOption.Default)
{
Console.WriteLine($"\nDownloading additional images:");
additionalImages = await DownloadImages(tle.list, Config.albumArtOption, tracks, organizer);
tracks?.AddRange(additionalImages);
}
if (tracks != null && tle.source.DownloadPath.Length > 0)
{
organizer.OrganizeAlbum(tracks, additionalImages);
}
m3uEditor.Update();
}
static async Task RunAlbumDownloads(TrackListEntry tle, FileManager organizer, List<Track> tracks, SemaphoreSlim semaphore, CancellationTokenSource cts)
{
var downloadTasks = tracks.Select(async track =>
{
await DownloadTask(tle, track, semaphore, organizer, cts, true, true, true);
});
await Task.WhenAll(downloadTasks);
}
static async Task OnAlbumSuccess(TrackListEntry tle, List<Track>? tracks)
{
if (tracks == null)
return;
var downloadedAudio = tracks.Where(t => !t.IsNotAudio && t.State == TrackState.Downloaded && t.DownloadPath.Length > 0);
if (downloadedAudio.Any())
@ -507,25 +593,41 @@ static partial class Program
}
}
List<Track>? additionalImages = null;
if (Config.albumArtOnly || succeeded && Config.albumArtOption != AlbumArtOption.Default)
static void OnAlbumFail(List<Track>? tracks)
{
Console.WriteLine($"\nDownloading additional images:");
additionalImages = await DownloadImages(tle.list, Config.albumArtOption, tracks);
tracks?.AddRange(additionalImages);
}
if (tracks == null || Config.IgnoreAlbumFail)
return;
if (tracks != null && tle.source.DownloadPath.Length > 0)
foreach (var track in tracks)
{
organizer.OrganizeAlbum(tracks, additionalImages);
if (track.DownloadPath.Length > 0 && File.Exists(track.DownloadPath))
{
try
{
if (Config.DeleteAlbumOnFail)
{
File.Delete(track.DownloadPath);
}
else if (Config.failedAlbumPath.Length > 0)
{
var newPath = Path.Join(Config.failedAlbumPath, Path.GetRelativePath(Config.parentDir, track.DownloadPath));
Directory.CreateDirectory(Path.GetDirectoryName(newPath));
Utils.Move(track.DownloadPath, newPath);
}
m3uEditor.Update();
Utils.DeleteAncestorsIfEmpty(Path.GetDirectoryName(track.DownloadPath), Config.parentDir);
}
catch (Exception e)
{
Printing.WriteLine($"Error: Unable to move or delete file '{track.DownloadPath}' after album fail: {e}");
}
}
}
}
static async Task<List<Track>> DownloadImages(List<List<Track>> downloads, AlbumArtOption option, List<Track>? chosenAlbum)
static async Task<List<Track>> DownloadImages(List<List<Track>> downloads, AlbumArtOption option, List<Track>? chosenAlbum, FileManager fileManager)
{
var downloadedImages = new List<Track>();
long mSize = 0;
@ -621,11 +723,11 @@ static partial class Program
bool allSucceeded = true;
var semaphore = new SemaphoreSlim(1);
var organizer = new FileManager();
foreach (var track in tracks)
{
await DownloadTask(null, track, semaphore, organizer, null, false, false);
using var cts = new CancellationTokenSource();
await DownloadTask(null, track, semaphore, fileManager, cts, false, false, false);
if (track.State == TrackState.Downloaded)
downloadedImages.Add(track);
@ -641,15 +743,12 @@ static partial class Program
}
static async Task DownloadTask(TrackListEntry? tle, Track track, SemaphoreSlim semaphore, FileManager organizer, CancellationTokenSource? cts, bool cancelOnFail, bool removeFromSource)
static async Task DownloadTask(TrackListEntry? tle, Track track, SemaphoreSlim semaphore, FileManager organizer, CancellationTokenSource cts, bool cancelOnFail, bool removeFromSource, bool organize)
{
if (track.State != TrackState.Initial)
return;
if (cts != null)
await semaphore.WaitAsync(cts.Token);
else
await semaphore.WaitAsync();
int tries = Config.unknownErrorRetries;
string savedFilePath = "";
@ -659,15 +758,15 @@ static partial class Program
{
await WaitForLogin();
cts?.Token.ThrowIfCancellationRequested();
cts.Token.ThrowIfCancellationRequested();
try
{
(savedFilePath, chosenFile) = await Search.SearchAndDownload(track, organizer);
(savedFilePath, chosenFile) = await Search.SearchAndDownload(track, organizer, cts);
}
catch (Exception ex)
{
WriteLine($"Exception thrown: {ex}", debugOnly: true);
WriteLine($"Error: {ex}", debugOnly: true);
if (!IsConnectedAndLoggedIn())
{
continue;
@ -682,13 +781,12 @@ static partial class Program
if (cancelOnFail)
{
cts?.Cancel();
cts.Cancel();
throw new OperationCanceledException();
}
}
else
{
WriteLine($"\n{ex.Message}\n{ex.StackTrace}\n", ConsoleColor.DarkYellow, true);
tries--;
continue;
}
@ -699,7 +797,7 @@ static partial class Program
if (tries == 0 && cancelOnFail)
{
cts?.Cancel();
cts.Cancel();
throw new OperationCanceledException();
}
@ -724,9 +822,9 @@ static partial class Program
}
}
if (track.State == TrackState.Downloaded && tle != null)
if (track.State == TrackState.Downloaded && organize)
{
organizer?.OrganizeAudio(tle, track, chosenFile);
organizer?.OrganizeAudio(track, chosenFile);
}
if (Config.onComplete.Length > 0)
@ -760,7 +858,13 @@ static partial class Program
}
}
WriteLine($"\nPrev [Up/p] / Next [Down/n] / Accept [Enter] / Accept & Exit Interactive [q] / Skip [Esc/s]\n", ConsoleColor.Green);
string retrieveAll1 = retrieveFolder ? "| [r] " : "";
string retrieveAll2 = retrieveFolder ? "| Load All Files " : "";
Console.WriteLine();
WriteLine($" [Up/p] | [Down/n] | [Enter] | [q] {retrieveAll1}| [Esc/s]", ConsoleColor.Green);
WriteLine($" Prev | Next | Accept | Accept & Quit Interactive {retrieveAll2}| Skip", ConsoleColor.Green);
Console.WriteLine();
while (true)
{
@ -770,17 +874,10 @@ static partial class Program
WriteLine($"[{aidx + 1} / {list.Count}]", ConsoleColor.DarkGray);
var folder = Utils.GreatestCommonDirectorySlsk(tracks.Select(t => t.FirstDownload.Filename));
if (retrieveFolder && !retrievedFolders.Contains(username + '\\' + folder))
{
Console.WriteLine("Getting all files in folder...");
await Search.CompleteFolder(tracks, response, folder);
retrievedFolders.Add(username + '\\' + folder);
}
PrintAlbum(tracks);
Console.WriteLine();
Loop:
string userInput = interactiveModeLoop().Trim();
switch (userInput)
{
@ -795,6 +892,24 @@ static partial class Program
case "q":
Config.interactiveMode = false;
return aidx;
case "r":
var folder = Utils.GreatestCommonDirectorySlsk(tracks.Select(t => t.FirstDownload.Filename));
if (retrieveFolder && !retrievedFolders.Contains(username + '\\' + folder))
{
Console.WriteLine("Getting all files in folder...");
int newFiles = await Search.CompleteFolder(tracks, response, folder);
retrievedFolders.Add(username + '\\' + folder);
if (newFiles == 0)
{
Console.WriteLine("No more files found.");
goto Loop;
}
else
{
Console.WriteLine($"Found {newFiles} more files in the folder:");
}
}
break;
case "":
return aidx;
}
@ -846,7 +961,9 @@ static partial class Program
}
else
{
if (!client.State.HasFlag(SoulseekClientStates.LoggedIn | SoulseekClientStates.LoggingIn | SoulseekClientStates.Connecting))
if (!client.State.HasFlag(SoulseekClientStates.LoggedIn)
&& !client.State.HasFlag(SoulseekClientStates.LoggingIn)
&& !client.State.HasFlag(SoulseekClientStates.Connecting))
{
WriteLine($"\nDisconnected, logging in\n", ConsoleColor.DarkYellow, true);
try { await Login(Config.useRandomLogin); }
@ -1017,7 +1134,7 @@ static partial class Program
public static bool IsConnectedAndLoggedIn()
{
return client != null && (client.State & (SoulseekClientStates.Connected | SoulseekClientStates.LoggedIn)) != 0;
return client != null && client.State.HasFlag(SoulseekClientStates.Connected) && client.State.HasFlag(SoulseekClientStates.LoggedIn);
}
}

50
slsk-batchdl/Search.cs
View file

@ -20,17 +20,17 @@ static class Search
public static RateLimitedSemaphore? searchSemaphore;
// very messy function that does everything
public static async Task<(string, SlFile?)> SearchAndDownload(Track track, FileManager organizer)
public static async Task<(string, SlFile?)> SearchAndDownload(Track track, FileManager organizer, CancellationTokenSource cts)
{
if (Config.DoNotDownload)
throw new Exception();
var responseData = new ResponseData();
IEnumerable<(SlResponse response, SlFile file)>? orderedResults = null;
var responseData = new ResponseData();
var progress = Printing.GetProgressBar(Config.displayMode);
var results = new SlDictionary();
var fsResults = new SlDictionary();
var cts = new CancellationTokenSource();
using var searchCts = new CancellationTokenSource();
var saveFilePath = "";
SlFile? chosenFile = null;
Task? downloadTask = null;
@ -62,7 +62,7 @@ static class Search
saveFilePath = organizer.GetSavePath(f.Filename);
fsUser = r.Username;
chosenFile = f;
downloadTask = Download.DownloadFile(r, f, saveFilePath, track, progress, cts);
downloadTask = Download.DownloadFile(r, f, saveFilePath, track, progress, cts, searchCts);
}
}
}
@ -111,7 +111,7 @@ static class Search
}
void onSearch() => Printing.RefreshOrPrint(progress, 0, $"Searching: {track}", true);
await RunSearches(track, results, getSearchOptions, responseHandler, cts.Token, onSearch);
await RunSearches(track, results, getSearchOptions, responseHandler, searchCts.Token, onSearch);
searches.TryRemove(track, out _);
searchEnded = true;
@ -142,7 +142,7 @@ static class Search
}
}
cts.Dispose();
searchCts.Dispose();
downloads:
@ -159,7 +159,7 @@ static class Search
try
{
downloading = 1;
await Download.DownloadFile(response, file, saveFilePath, track, progress);
await Download.DownloadFile(response, file, saveFilePath, track, progress, cts);
userSuccessCount.AddOrUpdate(response.Username, 1, (k, v) => v + 1);
return true;
}
@ -168,13 +168,17 @@ static class Search
chosenFile = null;
saveFilePath = "";
downloading = 0;
if (!IsConnectedAndLoggedIn())
throw;
Printing.WriteLine("Error: " + e.Message, ConsoleColor.DarkYellow, true);
userSuccessCount.AddOrUpdate(response.Username, -1, (k, v) => v - 1);
if (--trackTries <= 0)
{
Printing.RefreshOrPrint(progress, 0, $"Out of download retries: {track}", true);
Printing.WriteLine("Last error was: " + e.Message, ConsoleColor.DarkYellow, true);
Printing.WriteLine("Last error was: " + e.Message, ConsoleColor.DarkYellow);
throw new SearchAndDownloadException(FailureReason.OutOfDownloadRetries);
}
return false;
@ -284,7 +288,7 @@ static class Search
results.TryAdd(r.Username + "\\" + file.Filename, (r, file));
}
}
var cts = new CancellationTokenSource();
using var cts = new CancellationTokenSource();
await RunSearches(track, results, getSearchOptions, handler, cts.Token);
@ -427,7 +431,7 @@ static class Search
results.TryAdd(r.Username + "\\" + file.Filename, (r, file));
}
}
var cts = new CancellationTokenSource();
using var cts = new CancellationTokenSource();
await RunSearches(track, results, getSearchOptions, handler, cts.Token);
@ -461,10 +465,7 @@ static class Search
public static async Task<List<List<List<Track>>>> GetAggregateAlbums(Track track, ResponseData responseData)
{
int maxDiff = Config.necessaryCond.LengthTolerance;
if (maxDiff < 0)
maxDiff = 3;
int maxDiff = Config.aggregateLengthTol;
bool lengthsAreSimilar(int[] sorted1, int[] sorted2)
{
@ -555,13 +556,13 @@ static class Search
}
public static async Task<List<(string dir, SlFile file)>> GetAllFilesInFolder(string user, string folderPrefix)
public static async Task<List<(string dir, SlFile file)>> GetAllFilesInFolder(string user, string folderPrefix, CancellationToken? cancellationToken = null)
{
var browseOptions = new BrowseOptions();
var res = new List<(string dir, SlFile file)>();
folderPrefix = folderPrefix.TrimEnd('\\') + '\\';
var userFileList = await client.BrowseAsync(user, browseOptions);
var userFileList = await client.BrowseAsync(user, browseOptions, cancellationToken);
foreach (var dir in userFileList.Directories)
{
@ -572,22 +573,15 @@ static class Search
}
}
return res;
// It would be much better to use GetDirectoryContentsAsync. Unfortunately it only returns the file
// names without full paths, and DownloadAsync needs full paths in order to download files.
// Therefore it would not be possible to download any files that are in a subdirectory of the folder.
// var dir = await client.GetDirectoryContentsAsync(user, folderPrefix);
// var res = dir.Files.Select(x => (folderPrefix, x)).ToList();
// return res;
}
public static async Task CompleteFolder(List<Track> tracks, SearchResponse response, string folder)
public static async Task<int> CompleteFolder(List<Track> tracks, SearchResponse response, string folder, CancellationToken? cancellationToken = null)
{
int newFiles = 0;
try
{
var allFiles = await GetAllFilesInFolder(response.Username, folder);
var allFiles = await GetAllFilesInFolder(response.Username, folder, cancellationToken);
if (allFiles.Count > tracks.Count)
{
@ -599,6 +593,7 @@ static class Search
var fullPath = dir + '\\' + file.Filename;
if (!paths.Contains(fullPath))
{
newFiles++;
var newFile = new SlFile(file.Code, fullPath, file.Size, file.Extension, file.Attributes);
var t = new Track
{
@ -616,6 +611,7 @@ static class Search
{
Printing.WriteLine($"Error getting complete list of files: {ex}", ConsoleColor.DarkYellow);
}
return newFiles;
}
@ -633,7 +629,7 @@ static class Search
}
var groups = fileResponses
.GroupBy(inferTrack, new TrackComparer(ignoreCase: true, Config.necessaryCond.LengthTolerance))
.GroupBy(inferTrack, new TrackComparer(ignoreCase: true, Config.aggregateLengthTol))
.Select(x => (x, x.Select(y => y.Item1.Username).Distinct().Count()))
.Where(x => x.Item2 >= minShares)
.OrderByDescending(x => x.Item2)

6
slsk-batchdl/Test.cs
View file

@ -255,7 +255,7 @@ namespace Test
{
Config.input = strings[i];
Console.WriteLine(Config.input);
var res = await extractor.GetTracks(0, 0, false);
var res = await extractor.GetTracks(Config.input, 0, 0, false);
var t = res[0].list[0][0];
Assert(Extractors.StringExtractor.InputMatches(Config.input));
Assert(t.ToKey() == tracks[i].ToKey());
@ -268,7 +268,7 @@ namespace Test
{
Config.input = strings[i];
Console.WriteLine(Config.input);
var t = (await extractor.GetTracks(0, 0, false))[0].source;
var t = (await extractor.GetTracks(Config.input, 0, 0, false))[0].source;
Assert(Extractors.StringExtractor.InputMatches(Config.input));
Assert(t.ToKey() == albums[i].ToKey());
}
@ -317,7 +317,7 @@ namespace Test
};
var trackLists = new TrackLists();
trackLists.AddEntry(new TrackListEntry());
trackLists.AddEntry(new TrackListEntry(TrackType.Normal));
foreach (var t in notFoundInitial)
trackLists.AddTrackToLast(t);
foreach (var t in existingInitial)

27
slsk-batchdl/Utils.cs
View file

@ -89,10 +89,13 @@ public static class Utils
return path;
}
if (path.StartsWith("~"))
if (path.StartsWith('~'))
{
string homeDirectory = Environment.GetFolderPath(Environment.SpecialFolder.UserProfile);
return Path.Combine(homeDirectory, path.Substring(1).TrimStart('/').TrimStart('\\'));
path = Path.Join(homeDirectory, path.Substring(1).TrimStart('/').TrimStart('\\'));
if (path.Length > 0)
path = Path.GetFullPath(path);
}
return path;
@ -162,6 +165,26 @@ public static class Utils
}
}
public static void DeleteAncestorsIfEmpty(string startDir, string root)
{
string x = NormalizedPath(Path.GetFullPath(root));
string y = NormalizedPath(startDir);
if (x.Length == 0)
return;
while (y.StartsWith(x + '/') && FileCountRecursive(y) == 0)
{
Directory.Delete(y, true);
string prev = y;
y = NormalizedPath(Path.GetDirectoryName(y) ?? "");
if (prev.Length == y.Length)
break;
}
}
public static bool EqualsAny(this string input, string[] values, StringComparison comparison = StringComparison.Ordinal)
{
foreach (var value in values)