I am seeking advice regarding my ebook collection on a Linux system, which is stored on an external drive and sorted into categories. However, there are still many unsorted ebooks. I have tried using Calibre for organization, but it creates duplicate files during import on my main drive where I don’t want to keep any media. I would like to:
- Use Calibre’s automatic organization (tags, etc.) without duplicating files
- Maintain my existing folder structure while using Calibre
- Automatically sort the remaining ebooks into my existing categories/folder structure
I am considering the use of symlinks if there is a simple way to automate the process due to my large collection.
Regarding automatic sorting by category, I am looking for a solution that doesn’t require manual organization or a significant time investment. I’m wondering if there’s a way to extract metadata based on file hashes or any other method that doesn’t involve manual work. Most of the files should have title and author metadata, but some won’t. I am not in a rush to solve this issue since I can still locate most ebooks by their title without any organization.
Has anyone encountered a similar problem and found a solution? I would appreciate any suggestions for tools, scripts, or workflows that might help. Thank you in advance for any advice!
Migrate to Calibre and use Calibre Virtual Libraries. However based on the comments I’m reading, it looks like you want something that is not application based. Good luck with that.
If the files are literally duplicated (exact same bytes in the files, so matching md5sums) then maybe you could just delete the duplicates and maybe replace them with links.
Automatically sorting books by category isn’t so easy. Is the metadata any good? Are there categories already? ISBN’s? Even titles and authors? It starts to be kind of a project but you could possibly import MARC records (library metadata) which have some of thatinfo in them, if you can match up the books to library records. I expect that the openlibrary.org API still works but I haven’t used it in ages.
deleted by creator
I hope someone gives you a good answer, because I’d like one myself. My method has just been to do this stuff little by little. I would also recommend calibre web for interfacing instead of calibre. You can run both in docker, and access calibre on your server from whatever computer you happen to be on. I find centralizing collections makes the task of managing them at least more mentally manageable.
You might want to give an idea of the size of your library. What some people consider large, others might consider nothing much. If it is exceedingly large you’re better off asking someplace with more data hoarders instead of a general Linux board.
I hope someone gives you a good answer
I honestly don’t know that there is one. What OP is looking for is effectively an AI librarian… this is literally a full-time job for some people. I’m sure OP doesn’t have quite that many books, but the point remains
How many ebooks are you talking about (millions)? Is there just a question of finding duplicated files? That’s easy with a shell script. For metadata, see if the books already have it since a lot do. After that, you can use fairly crude hacks as an initial pass at matching library records. There’s code like that around already, try some web searches, maybe code4lib (library related programming) if that is still around. I saw your earlier comment before you deleted it and it was perfectly fine.
I tried to ingest a four terabyte epub library once. Even getting the data ingested with the author and title in the right spot was almost impossible. If a duplicates weren’t just slightly wrong would be a different story but the duplicates are often misspells or different spellings.
Realistically the best thing you can do is get an output of file name, title, author and hand dedupe, but even then you’re going to have to be careful about quality and language and all kinds of other strange issues you run into with large libraries.
In the end I gave up and only stored what I really wanted and would realistically ever need and that was small enough to hand cull.
When I had to match against misspells I found Levenshtein distance to be most useful.
https://en.wikipedia.org/wiki/Levenshtein_distance?wprov=sfla1
Yeah, I wrote some python once to give me voice control over a plex server. Distance algorithms did okay I had a lot better results out of fuzzy wuzzy though
Kind of a dicey prospect with risk when you’re doing duplication though.
Then you run into problems with things like Harry Potter and the philosopher’s Stone versus Harry Potter and the sorcerer’s Stone. Depending on how badly your database is degenerated you can even end up with words out of order and all kinds of crazy crap. If the database is truly large just truing the database up can be unreasonably time-consuming.
I was pretty amazed at all the different versions of string search and comparison algorithms.
I’ve run into exactly the same issue with my large ttrpg ebook/pdf collection (+100k file data hoarding… it’s not a problem, I swear) and I’ve not really found a good option I’m entirely happy with. Calibre duplicates everything and I don’t like the thought of having my collection’s organization tied to a specific piece of software if I just delete my duplicates. Plus I’m elitist and think the UI/logo are gross to look at.
Zotero is the least worst option I’ve found, but it’s geared towards scholarly journals and such, so not great, but serviceable. Not sure if it’s on linux though.
Jellyfin is apparently able to handle ebooks with a plugin, though I didn’t particularly care for it when I tried it months ago.
There’s a handful of other ebook software out there, mostly geared towards comics/manga, so depending on what you have those might be worth looking for.
I’d like to use Obsidian for it and just turn the directory into a vault and let it automatically scan the folders for files, but that doesn’t work great either.
The best piece of software I’ve seen that could potentially handle it is an app called Stashapp… which is unfortunately geared towards adult film. But it’s feature-set if it could be applied to PDFs seems like it would be ideal.
Zotero is on Linux, and it has a LibreOffice plugin as well. Though, I do not like Zotero. Zotero is more geared towards reference management, but it also offers some pdf, epub management. But I find their document management too tedious. It’s just easier for me to just rename files. That served me well for a long time.
Yeah, I’ve tried, both for actual files and for tracking my reading across multiple platforms, and nothing really seems to fit my needs, especially when I want to actually read them on an Android ereader. Anything I choose seems to have a lot of manual effort, frequently, or just a dumpster fire of an actual reading experience.
I feel like I’m eventually going to have to make my own, which is fine, I guess, but I’m definitely not comfortable actually managing a community project or just building up the codebase or documentation to the level someone else would be enthusiastic to use as a jumping off point to manage themselves, so it will probably just stay a personal project that ends up not helping anyone else solve the same problems I have.
Distilbert https://huggingface.co/distilbert/distilroberta-base
…was setup for something like that here but note that the repo that runs this has an “unsafe” warning that I have not looked into: https://huggingface.co/spaces/nasrin2023ripa/multilabel-book-genre-classifier
https://huggingface.co/spaces/nasrin2023ripa/multilabel-book-genre-classifier/tree/main
It might be fine or whatnot, I’m on mobile and can’t see the file in question. The associated Python code might be a helpful starting point.
In my experience, most models intentionally obfuscate copyright sources. They all know the materials to various degrees, but they are not intended to replicate sources. They all have strong interference in place to obscure both their recognition and reproduction potential. If, for instance, you can identify where errors are inserted and make a few corrections, they often continue adding a few details that are from the original source. If this is done a few times in a row, they tend to gain more freedom before reverting to obfuscation again. This is the behavior I look out for. It is a strong tool too if you get creative in application.
Perhaps someone posts an API to look up the library of congress classification of a work based on a few lines or something. GL
Why is duplicate files a problem?
I don’t like keeping duplicate files, especially in my main drive where I don’t store media. If I didn’t mind duplicate files, it wouldn’t be an issue.
I guess I’m a little confused… why are there duplicates at all? What operation are you performing that ends with duplicates?
Is this drive just where you download them to, and then move them to your organized drive? Why do the books ever touch this drive at all? If there isn’t supposed to be media on the drive, why not just delete the source folder after the organization task is complete?
In regards to another reply, there are multiple options of dealing with duplicates in calibre. From merging to deleting them.
It’s been a while since I had to do any library setup with calibre and Im kinda confused by what you are saying with “but it duplicates files when importing in a different drive where I don’t want ebooks”
Do you have multiple libraries in calibre?
Are you using multiple library managers?
Why not just use calibres folder structure?
And also if you don’t know about it - mobileread forums has been around a long time and has a whole section on calibre where you could probably get more specific help.
