Dataset Tools for Imaging and Captioning

V0.58.5
duskfallcrew
2 months ago

Currently we'll pull big updates both here https://github.com/duskfallcrew/Dataset-Tools
And in civitai.
This isn't perfect yet, it's a bit backwards compared to the original, but we're paying for claude to help fix this

Dataset-Tools: Enhancing Dataset Visualization and Text Editing

Dataset-Tools is a versatile script designed to assist in visualizing datasets and editing associated text files. Developed using PyQt6, it provides an intuitive graphical interface for managing images and their corresponding annotations. Inspired by tools within the Civitai community, this project aims to empower users in enhancing their dataset management capabilities.

You're more than welcome to help develop this tool, I am not really a programmer, I am aware of this - I am doing tiny things with Chat GPT to further help my community! I'm also aware there are people I may have spoken to over the months that have indeed developed something similar, I am not the first one to make this, nor am i pulling a JoJo Siwa xD.

How to Use Dataset-Tools

Launching the Application

Ensure Python is installed on your system. Clone or download the Dataset-Tools repository from GitHub. Setting Up Environment

Install necessary dependencies using pip:

Copy code pip install PyQt6 Pillow Running the Application

Run the application using Python: css Copy code python main.py

User Interface Overview

Main Window: The application window opens with options to view images, edit associated text, and select themes.

Image Display: Images are displayed in the top section of the window with options for scaling and centering. Text Editor: Below the image, there's a text box for editing and saving text associated with the selected image. Listboxes: Lists images and text files available in the current directory. Buttons: Includes "Save" to save edited text, "Close" to exit the application gracefully, and "Select Image" to pick an image for editing. Managing Images and Text

Selecting Images: Click on an image in the listbox or gallery to display it in the main window.

Editing Text: Edit the text in the text box. Use the "Save" button to save changes to the associated text file. Changing Themes

Theme Selection: Use the dropdown menu labeled "Theme Selection" to choose from available themes.

Applying Themes: Themes change the background, text colors, and button styles to suit different preferences. Customizing Themes (Advanced)

Editing Themes: Modify or add themes in the themes dictionary within the main.py file. Each theme consists of background color (bg), foreground color (fg), text background (text_bg), text foreground (text_fg), button background (button_bg), and button foreground (button_fg).

Overview of Themes

Dataset-Tools offers a range of themes to customize the interface according to user preferences:

Beetlejuice Inspired Light Theme Dark Theme Pastel V2 Night Sky Fire Red Sunset Orange Lavender Ocean Blue Mint Forest Green Some themes, like Pastel V1, are currently disabled due to issues with text colors, which are being addressed in ongoing development.

Key Features

Graphical User Interface (GUI): Designed initially with Tkinter and later migrated to PyQt6 for improved functionality and aesthetics. Image Gallery: Allows for browsing and selection of images within the dataset. Text Editing: Supports real-time editing and saving of associated text files. Theme Customization: Choose from various predefined themes to personalize the application's appearance.

Future Developments

Educational Resources: Planned video tutorials and comprehensive text guides to aid users in navigating Dataset-Tools effectively. Enhanced Theme Functionality: Ongoing improvements to themes, focusing on legibility and aesthetic appeal across different datasets. Deployment and Testing: Targeting broader compatibility across operating systems and screen resolutions to optimize user experience.

About the Creator

Dataset-Tools is developed by the Duskfall Portal Crew, a diverse system navigating life with DID, ADHD, Autism, and CPTSD. Our motivation stems from leveraging AI to promote inclusivity and mental health awareness. Join us in exploring identity and creative expression through technology.

Join Our Community

Website: Earth & Dusk Portal Discord: Join our Discord Backups: Hugging Face Support Us Send a Pizza: Buy us a pizza Subreddit: Reddit Community Thank you for supporting us on our journey and contributing to the Earth and Dusk community.

Credits

ChatGPT 3.5 & 4o: Powering innovative solutions and creative endeavors. Canvas icons created by Freepik - Flaticon: Enhancing visual appeal and functionality.

Sample Images

Here you can see the themes and gui overlook.

GUI Overview

Screenshot 2024-06-15 at 19 09 31

Sample Themes

Screenshot 2024-06-15 at 19 09 40Screenshot 2024-06-15 at 19 09 48Screenshot 2024-06-15 at 19 10 27Screenshot 2024-06-15 at 19 10 36

Errors to Fix

Screenshot 2024-06-15 at 19 11 19Screenshot 2024-06-15 at 19 11 31

Changelog

Started with Tkinter Gui, but moved to PYQT6.

Realized it needed more to it.

Added gallery, text editing, image preview.

Text box list.

Save, Close and Select Image.

Commented out themes that don't work.

Tried to add an icon.

Fixing aspect ratio is ... well i'm on a retina 5k screen, so go figure. - This is because if you select a TALLER image it'll break and you can't close it or edit the text, i'm working on it.

The OLDER file in this specific repo, as i'm finally merging to main will give you back some stuff btue the images are tiny.

Indentation issues are fixed.

What the OLD FILE Looks like - and that's ok!

Screenshot 2024-06-15 at 16 56 14

Coming Soon

Video tutorial via youtube.

Tutorial in text, images.

Fixing themes where the text is just attrocious. That's my fault, sorry.

When we're out of MAIN development mode, i want to try and get this as a homebrew cask, and figure out how to all that - we're just not ready yet.

Get people to test it on different OS boxes, so we can decide WHAT sizing is best.

About the Creator:

We are the Duskfall Portal Crew, a DID system with over 300 alters, navigating life with DID, ADHD, Autism, and CPTSD. We believe in AI’s potential to break down barriers and enhance mental health, despite its challenges. Join us on our creative journey exploring identity and expression.

Join Our Community:

Website:

End Media

Discord:

Join our Discord

Backups:

Hugging Face

Support Us:

Send a Pizza

Subreddit:

Reddit

Thank you for being part of our journey and supporting the Earth and Dusk community.

the "USE WITHOUT CREDITING" is only unticked for those who want to fork, it's not that i demand credit, it's that I would like to learn more about how this program works how i've made it and how changes are applied.

Credits

ChatGPT 3.5 & 4o

Support of my peers, and the community at Large..

Canvas icons created by Freepik - Flaticon

Where we started from over 24 hours ago:

Screenshot 2024-06-14 at 22 00 40Screenshot 2024-06-14 at 22 09 01Screenshot 2024-06-15 at 00 03 47Screenshot 2024-06-15 at 00 03 55Screenshot 2024-06-15 at 00 04 16Screenshot 2024-06-15 at 14 06 00Screenshot 2024-06-15 at 15 01 45

Read more...
Download (2.26 MB) Download available on desktop only

Popularity

90 ~10

Info

Base model: Other

Version V0.58.5: 1 File

To download these files, please visit this page from a desktop computer.

About this version: V0.58.5

PR: Major Overhaul - Parsing, Stability, UI, Security, and Docs

This pull request introduces a significant set of improvements and features, representing a substantial leap forward in the development of Dataset-Tools. It focuses on enhancing core stability, overhauling the metadata parsing pipeline, adding key usability features like drag-and-drop, integrating new custom parsers, and updating project documentation for clarity and accuracy.

⚑ Key Highlights & New Features

  • Drag and Drop Support: Users can now drag and drop image/model files or entire folders directly onto the main application window for easier loading. Dropped single files are auto-selected.

  • New Custom Parsers:

    • RuinedFooocusFormat: Added support for RuinedFooocus JSON metadata.

    • CivitaiComfyUIFormat: Added support for Civitai's ComfyUI-based UserComment metadata, including mojibake decoding.

    • Enabled GGUFParser: Basic metadata extraction from .gguf model files is now functional.

  • Major Stability Improvements: Addressed numerous critical bugs related to imports, attribute access, and type errors across the application.

  • Enhanced Vendored Code: Significantly refactored the adapted sd-prompt-reader components for better status handling, error reporting, and integration.

  • Security Hardening: Fixed XML parsing vulnerabilities by integrating defusedxml.

  • Improved Code Quality: Applied extensive linting fixes using Ruff, adhering more closely to PEP 8 and improving maintainability.

  • Updated Documentation: README.md and SECURITY.MD have been overhauled; NOTICE.md added for vendored code attribution.

πŸ› Core Stability & Bug Fixes

  • Resolved critical import errors (ModuleNotFoundError, NameError) across ui.py, logger.py, correct_types.py, and vendored_sdpr/* by correcting import paths and resolving circular dependencies.

  • Fixed AttributeError: 'str' object has no attribute 'value' by refactoring EmptyField, UpField, DownField in correct_types.py to be proper enum.Enum classes. UI and parsers now correctly use .value and get_ordered_labels().

  • Fixed AttributeError in widgets.py related to ExtensionType by using correct attribute names (e.g., SCHEMA_FILES, MODEL_FILES).

  • Resolved TypeError for parser __init__ calls (e.g., for A1111) in ImageDataReader by ensuring passed arguments match parser signatures.

  • Fixed ValueError: not enough values to unpack in A1111._sd_format() by correcting regex match unpacking.

  • Addressed NameError: name 'Path' is not defined in ImageDataReader by adding the pathlib.Path import.

  • Eliminated F811 Redefinition of unused __init__ in access_disk.py.

  • Corrected F821 Undefined name traceback in access_disk.py by importing traceback.

βš™οΈ Vendored SD-Prompt-Reader Enhancements & Fixes

  • Refactored ImageDataReader._try_parser() helper for clearer instantiation and more robust status checking of individual format parsers.

  • Ensured ImageDataReader.status is accurately updated based on the success/failure of sub-parsers, resolving previously misleading FORMAT_ERROR reports.

  • Improved status and error message handling within individual format parsers (e.g., A1111._process() now reliably sets its own success/failure status).

  • Standardized logger names within the vendored_sdpr components to use the "DSVendored_SDPR" prefix.

  • Pruned vendored_sdpr/format/utility.py to essential functions, removing unused GUI dependencies.

  • Integrated defusedxml for safer XML parsing in ImageDataReader (for DrawThings XMP data), addressing Bandit-flagged vulnerabilities (B408, B318). Added defusedxml as a project dependency.

✨ New Parser Integrations

(Covered in Key Highlights)

🎨 Code Quality & Linting (Ruff)

Applied extensive linting fixes across the codebase using Ruff (ruff format . and ruff check . --fix):

  • Resolved numerous E701, E702, E703 style issues (multiple statements per line, semicolons).

  • Removed many F401 (unused imports) and F841 (unused local variables) after careful review.

  • Corrected F403/F405 related to star imports.

  • Fixed E741 (ambiguous variable names).

  • Resolved E722 (bare except).

  • Corrected F541 (f-strings without placeholders).

These changes significantly improve code readability, maintainability, and adherence to PEP 8.

πŸ”Š Logging System Improvements

  • Resolved the WARNING in main.py regarding logger reconfiguration by aligning the function call to reconfigure_all_loggers. CLI log level setting is now fully effective.

  • Enhanced debug logging in key areas for better diagnostics.

πŸ“š Documentation & Project Files

  • README.md: Overhauled with details on new features (drag & drop, new parsers), revised installation instructions (reflecting pyproject.toml), updated license information, and improved overall structure and formatting.

  • SECURITY.md: Updated to be more user-friendly, clearly state the project's GPL-3.0 license, define support for the current main branch, and list security tools now in use (pip-audit, Bandit, Ruff).

  • NOTICE.md: Created to provide proper attribution and license information for vendored code, particularly the MIT-licensed components from sd-prompt-reader.

πŸ”¬ Testing Notes (Brief)

  • Manual testing performed for A1111/Forge PNGs, showing correct parsing and status reporting.

  • Drag and drop functionality tested with single files and folders.

  • Civitai ComfyUI JPGs and RuinedFooocus JPEGs now correctly identified by their respective parsers.

  • (Add any other key tests you performed).

πŸ‘€ Areas for Particular Review (Optional)

  • Verification of __init__ signatures for NovelAI(info=...) and ComfyUI(info=...) calls within ImageDataReader.py (regarding width/height arguments) to ensure they match the actual definitions in those parsers.

  • The robustness of the regex in metadata_parser.py's make_paired_str_dict for diverse A1111 settings strings (this is an ongoing refinement).

  • The refactored ComfyUI graph traversal logic in vendored_sdpr/format/comfyui.py for correctness and completeness with varied workflows.

https://github.com/Ktiseos-Nyx/Dataset-Tools/releases/tag/0.58.5

7 Versions

πŸ˜₯ There are no Dataset Tools for Imaging and Captioning V0.58.5 prompts yet!

Go ahead and upload yours!

No results

Your query returned no results – please try removing some filters or trying a different term.