Reference

Configuration

KToolBox Configuration

Attributes:

Name	Type	Description
api	APIConfiguration	Kemono API Configuration
downloader	DownloaderConfiguration	File Downloader Configuration
job	JobConfiguration	Download jobs Configuration
logger	LoggerConfiguration	Logger configuration
ssl_verify	bool	Enable SSL certificate verification for Kemono API server and download server
json_dump_indent	int	Indent of JSON file dump
use_uvloop	bool	Use uvloop for asyncio (Disabled on Windows by default) uvloop will improve concurrent performance, but it is not compatible with Windows. Install uvloop by pip install ktoolbox[uvloop] or it will not work.

Source code in ktoolbox/configuration.py

class Configuration(BaseSettings):
    # noinspection SpellCheckingInspection,GrazieInspection
    """
    KToolBox Configuration

    :ivar api: Kemono API Configuration
    :ivar downloader: File Downloader Configuration
    :ivar job: Download jobs Configuration
    :ivar logger: Logger configuration
    :ivar ssl_verify: Enable SSL certificate verification for Kemono API server and download server
    :ivar json_dump_indent: Indent of JSON file dump
    :ivar use_uvloop: Use uvloop for asyncio (Disabled on Windows by default) \
    uvloop will improve concurrent performance, but it is not compatible with Windows. \
    Install uvloop by `pip install ktoolbox[uvloop]` or it will not work.
    """
    api: APIConfiguration = APIConfiguration()
    downloader: DownloaderConfiguration = DownloaderConfiguration()
    job: JobConfiguration = JobConfiguration()
    logger: LoggerConfiguration = LoggerConfiguration()

    ssl_verify: bool = True
    json_dump_indent: int = 4
    use_uvloop: bool = True

    # noinspection SpellCheckingInspection
    model_config: ClassVar[SettingsConfigDict] = SettingsConfigDict(
        env_prefix='ktoolbox_',
        env_nested_delimiter='__',
        env_file=['.env', 'prod.env'],
        env_file_encoding='utf-8',
        extra='ignore'
    )

api: APIConfiguration = APIConfiguration() class-attribute instance-attribute

downloader: DownloaderConfiguration = DownloaderConfiguration() class-attribute instance-attribute

job: JobConfiguration = JobConfiguration() class-attribute instance-attribute

logger: LoggerConfiguration = LoggerConfiguration() class-attribute instance-attribute

ssl_verify: bool = True class-attribute instance-attribute

json_dump_indent: int = 4 class-attribute instance-attribute

use_uvloop: bool = True class-attribute instance-attribute

APIConfiguration

Kemono API Configuration

Attributes:

Name	Type	Description
scheme	Literal['http', 'https']	Kemono API URL scheme
netloc	str	Kemono API URL netloc
statics_netloc	str	URL netloc of Kemono server for static files (e.g. images)
files_netloc	str	URL netloc of Kemono server for post files
path	str	Kemono API URL root path
timeout	float	API request timeout
retry_times	int	API request retry times (when request failed)
retry_interval	float	Seconds of API request retry interval
session_key	str	Session key that can be found in cookies after a successful login

Source code in ktoolbox/configuration.py

class APIConfiguration(BaseModel):
    """
    Kemono API Configuration

    :ivar scheme: Kemono API URL scheme
    :ivar netloc: Kemono API URL netloc
    :ivar statics_netloc: URL netloc of Kemono server for static files (e.g. images)
    :ivar files_netloc: URL netloc of Kemono server for post files
    :ivar path: Kemono API URL root path
    :ivar timeout: API request timeout
    :ivar retry_times: API request retry times (when request failed)
    :ivar retry_interval: Seconds of API request retry interval
    :ivar session_key: Session key that can be found in cookies after a successful login
    """
    scheme: Literal["http", "https"] = "https"
    netloc: str = "kemono.cr"
    statics_netloc: str = "img.kemono.cr"
    files_netloc: str = "kemono.cr"
    path: str = "/api/v1"
    timeout: float = 5.0
    retry_times: int = 3
    retry_interval: float = 2.0
    session_key: str = ""

scheme: Literal['http', 'https'] = 'https' class-attribute instance-attribute

netloc: str = 'kemono.cr' class-attribute instance-attribute

statics_netloc: str = 'img.kemono.cr' class-attribute instance-attribute

files_netloc: str = 'kemono.cr' class-attribute instance-attribute

path: str = '/api/v1' class-attribute instance-attribute

timeout: float = 5.0 class-attribute instance-attribute

retry_times: int = 3 class-attribute instance-attribute

retry_interval: float = 2.0 class-attribute instance-attribute

session_key: str = '' class-attribute instance-attribute

DownloaderConfiguration

File Downloader Configuration

Attributes:

Name	Type	Description
scheme	Literal['http', 'https']	Downloader URL scheme
timeout	float	Downloader request timeout
encoding	str	Charset for filename parsing and post content text saving
buffer_size	int	Number of bytes of file I/O buffer for each downloading file
chunk_size	int	Number of bytes of chunk of downloader stream
temp_suffix	str	Temp filename suffix of downloading files
retry_times	int	Downloader retry times (when download failed)
retry_stop_never	bool	Never stop downloader from retrying (when download failed) (retry_times will be ignored when enabled)
retry_interval	float	Seconds of downloader retry interval
tps_limit	float	Maximum connections established per second
use_bucket	bool	Enable local storage bucket mode
bucket_path	Path	Path of local storage bucket
reverse_proxy	str	Reverse proxy format for download URL. Customize the filename format by inserting an empty {} to represent the original URL. For example: https://example.com/{} will be https://example.com/https://n1.kemono.su/data/66/83/xxxxx.jpg; https://example.com/?url={} will be https://example.com/?url=https://n1.kemono.su/data/66/83/xxxxx.jpg

Source code in ktoolbox/configuration.py

class DownloaderConfiguration(BaseModel):
    """
    File Downloader Configuration

    :ivar scheme: Downloader URL scheme
    :ivar timeout: Downloader request timeout
    :ivar encoding: Charset for filename parsing and post content text saving
    :ivar buffer_size: Number of bytes of file I/O buffer for each downloading file
    :ivar chunk_size: Number of bytes of chunk of downloader stream
    :ivar temp_suffix: Temp filename suffix of downloading files
    :ivar retry_times: Downloader retry times (when download failed)
    :ivar retry_stop_never: Never stop downloader from retrying (when download failed) \
    (``retry_times`` will be ignored when enabled)
    :ivar retry_interval: Seconds of downloader retry interval
    :ivar tps_limit: Maximum connections established per second
    :ivar use_bucket: Enable local storage bucket mode
    :ivar bucket_path: Path of local storage bucket
    :ivar reverse_proxy: Reverse proxy format for download URL. \
    Customize the filename format by inserting an empty ``{}`` to represent the original URL. \
    For example: ``https://example.com/{}`` will be ``https://example.com/https://n1.kemono.su/data/66/83/xxxxx.jpg``;  \
    ``https://example.com/?url={}`` will be ``https://example.com/?url=https://n1.kemono.su/data/66/83/xxxxx.jpg``
    """
    scheme: Literal["http", "https"] = "https"
    timeout: float = 30.0
    encoding: str = "utf-8"
    buffer_size: int = 20480
    chunk_size: int = 1024
    temp_suffix: str = "tmp"
    retry_times: int = 10
    retry_stop_never: bool = False
    retry_interval: float = 3.0
    tps_limit: float = 1.0
    use_bucket: bool = False
    bucket_path: Path = Path("./.ktoolbox/bucket_storage")
    reverse_proxy: str = "{}"

    @model_validator(mode="after")
    def check_bucket_path(self) -> "DownloaderConfiguration":
        if self.use_bucket:
            # noinspection PyBroadException
            try:
                bucket_path = Path(self.bucket_path)
                bucket_path.mkdir(parents=True, exist_ok=True)
                with tempfile.TemporaryFile(dir=bucket_path) as temp_file:
                    temp_link_file_path = f"{bucket_path / temp_file.name}.hlink"
                    os.link(temp_file.name, temp_link_file_path)
                    os.remove(temp_link_file_path)
            except Exception:
                self.use_bucket = False
                logger.exception(f"`DownloaderConfiguration.bucket_path` is not available, "
                                 f"`DownloaderConfiguration.use_bucket` has been disabled.")
        return self

scheme: Literal['http', 'https'] = 'https' class-attribute instance-attribute

timeout: float = 30.0 class-attribute instance-attribute

encoding: str = 'utf-8' class-attribute instance-attribute

buffer_size: int = 20480 class-attribute instance-attribute

chunk_size: int = 1024 class-attribute instance-attribute

temp_suffix: str = 'tmp' class-attribute instance-attribute

retry_times: int = 10 class-attribute instance-attribute

retry_stop_never: bool = False class-attribute instance-attribute

retry_interval: float = 3.0 class-attribute instance-attribute

tps_limit: float = 1.0 class-attribute instance-attribute

use_bucket: bool = False class-attribute instance-attribute

bucket_path: Path = Path('./.ktoolbox/bucket_storage') class-attribute instance-attribute

reverse_proxy: str = '{}' class-attribute instance-attribute

check_bucket_path()

Source code in ktoolbox/configuration.py

@model_validator(mode="after")
def check_bucket_path(self) -> "DownloaderConfiguration":
    if self.use_bucket:
        # noinspection PyBroadException
        try:
            bucket_path = Path(self.bucket_path)
            bucket_path.mkdir(parents=True, exist_ok=True)
            with tempfile.TemporaryFile(dir=bucket_path) as temp_file:
                temp_link_file_path = f"{bucket_path / temp_file.name}.hlink"
                os.link(temp_file.name, temp_link_file_path)
                os.remove(temp_link_file_path)
        except Exception:
            self.use_bucket = False
            logger.exception(f"`DownloaderConfiguration.bucket_path` is not available, "
                             f"`DownloaderConfiguration.use_bucket` has been disabled.")
    return self

PostStructureConfiguration

Post path structure model

• Default:

  ..
  ├─ content.txt
  ├─ <Post file>
  ├─ <Post data (post.json)>
  └─ attachments
     ├─ 1.png
     └─ 2.png
  

• Available properties for file

  Property	Type
  id	String
  user	String
  service	String
  title	String
  added	Date
  published	Date
  edited	Date

Attributes:

Name	Type	Description
attachments	Path	Sub path of attachment directory
content	Path	Sub path of post content file
content_filepath	Path	(Deprecated, Use content instead) Sub path of post content file
file	str	The format of the post file filename (file is not attachment, each post has only one file, usually the cover image) Customize the filename format by inserting an empty {} to represent the basic filename. You can use some of the properties in Post. For example: {title}_{} could result in filenames like TheTitle_Stelle_lv5_logo.gif, TheTitle_ScxHjZIdxt5cnjaAwf3ql2p7.jpg, etc.

Source code in ktoolbox/configuration.py

class PostStructureConfiguration(BaseModel):
    # noinspection SpellCheckingInspection
    """
    Post path structure model

    - Default:
    ```
    ..
    ├─ content.txt
    ├─ <Post file>
    ├─ <Post data (post.json)>
    └─ attachments
       ├─ 1.png
       └─ 2.png
    ```

    - Available properties for ``file``

        | Property      | Type   |
        |---------------|--------|
        | ``id``        | String |
        | ``user``      | String |
        | ``service``   | String |
        | ``title``     | String |
        | ``added``     | Date   |
        | ``published`` | Date   |
        | ``edited``    | Date   |

    :ivar attachments: Sub path of attachment directory
    :ivar content: Sub path of post content file
    :ivar content_filepath: (**Deprecated**, Use ``content`` instead) Sub path of post content file
    :ivar file: The format of the post `file` filename (`file` is not `attachment`, each post has only one `file`, usually the cover image) \
    Customize the filename format by inserting an empty ``{}`` to represent the basic filename. \
    You can use some of the [properties][ktoolbox.configuration.JobConfiguration] \
    in Post. For example: ``{title}_{}`` could result in filenames like \
    ``TheTitle_Stelle_lv5_logo.gif``, ``TheTitle_ScxHjZIdxt5cnjaAwf3ql2p7.jpg``, etc.
    """
    attachments: Path = Path("attachments")
    content: Path = Path("content.txt")
    content_filepath: Path = Path("content.txt")
    file: str = "{id}_{}"

    @field_validator("content_filepath")
    def content_filepath_validator(cls, v):
        # noinspection PyUnresolvedReferences
        if v != cls.model_fields["content_filepath"].default:
            warnings.warn(
                "`PostStructureConfiguration.content_filepath` is deprecated and is scheduled for removal in further version. "
                "Use `PostStructureConfiguration.content` instead",
                FutureWarning
            )

attachments: Path = Path('attachments') class-attribute instance-attribute

content: Path = Path('content.txt') class-attribute instance-attribute

content_filepath: Path = Path('content.txt') class-attribute instance-attribute

file: str = '{id}_{}' class-attribute instance-attribute

content_filepath_validator(v)

Source code in ktoolbox/configuration.py

@field_validator("content_filepath")
def content_filepath_validator(cls, v):
    # noinspection PyUnresolvedReferences
    if v != cls.model_fields["content_filepath"].default:
        warnings.warn(
            "`PostStructureConfiguration.content_filepath` is deprecated and is scheduled for removal in further version. "
            "Use `PostStructureConfiguration.content` instead",
            FutureWarning
        )

JobConfiguration

Download jobs Configuration

• Available properties for post_dirname_format and filename_format

  Property	Type
  id	String
  user	String
  service	String
  title	String
  added	Date
  published	Date
  edited	Date

Attributes:

Name	Type	Description
count	int	Number of coroutines for concurrent download
post_dirname_format	str	Customize the post directory name format, you can use some of the properties in Post. e.g. [{published}]{id} > [2024-1-1]123123, {user}_{published}_{title} > 234234_2024-1-1_TheTitle
post_structure	PostStructureConfiguration	Post path structure
mix_posts	bool	Save all files from different posts at same path in creator directory. It would not create any post directory, and CreatorIndices would not been recorded.
sequential_filename	bool	Rename attachments in numerical order, e.g. 1.png, 2.png, ...
filename_format	str	Customize the filename format by inserting an empty {} to represent the basic filename. Similar to post_dirname_format, you can use some of the properties in Post. For example: {title}_{} could result in filenames like TheTitle_b4b41de2-8736-480d-b5c3-ebf0d917561b, TheTitle_af349b25-ac08-46d7-98fb-6ce99a237b90, etc. You can also use it with sequential_filename. For instance, [{published}]_{} could result in filenames like [2024-1-1]_1.png, [2024-1-1]_2.png, etc.
allow_list	Set[str]	Download files which match these patterns (Unix shell-style), e.g. ["*.png"]
block_list	Set[str]	Not to download files which match these patterns (Unix shell-style), e.g. ["*.psd","*.zip"]

Source code in ktoolbox/configuration.py

class JobConfiguration(BaseModel):
    """
    Download jobs Configuration

    - Available properties for ``post_dirname_format`` and ``filename_format``

        | Property      | Type   |
        |---------------|--------|
        | ``id``        | String |
        | ``user``      | String |
        | ``service``   | String |
        | ``title``     | String |
        | ``added``     | Date   |
        | ``published`` | Date   |
        | ``edited``    | Date   |

    :ivar count: Number of coroutines for concurrent download
    :ivar post_dirname_format: Customize the post directory name format, you can use some of the \
    [properties][ktoolbox.configuration.JobConfiguration] in ``Post``. \
    e.g. ``[{published}]{id}`` > ``[2024-1-1]123123``, ``{user}_{published}_{title}`` > ``234234_2024-1-1_TheTitle``
    :ivar post_structure: Post path structure
    :ivar mix_posts: Save all files from different posts at same path in creator directory. \
    It would not create any post directory, and ``CreatorIndices`` would not been recorded.
    :ivar sequential_filename: Rename attachments in numerical order, e.g. ``1.png``, ``2.png``, ...
    :ivar filename_format: Customize the filename format by inserting an empty ``{}`` to represent the basic filename.
    Similar to post_dirname_format, you can use some of the [properties][ktoolbox.configuration.JobConfiguration] \
    in Post. For example: ``{title}_{}`` could result in filenames like \
    ``TheTitle_b4b41de2-8736-480d-b5c3-ebf0d917561b``, ``TheTitle_af349b25-ac08-46d7-98fb-6ce99a237b90``, etc. \
    You can also use it with ``sequential_filename``. For instance, \
    ``[{published}]_{}`` could result in filenames like ``[2024-1-1]_1.png``, ``[2024-1-1]_2.png``, etc.
    :ivar allow_list: Download files which match these patterns (Unix shell-style), e.g. ``["*.png"]``
    :ivar block_list: Not to download files which match these patterns (Unix shell-style), e.g. ``["*.psd","*.zip"]``
    """
    count: int = 4
    post_dirname_format: str = "{title}"
    post_structure: PostStructureConfiguration = PostStructureConfiguration()
    mix_posts: bool = False
    sequential_filename: bool = False
    filename_format: str = "{}"
    # noinspection PyDataclass
    allow_list: Set[str] = Field(default_factory=set)
    # noinspection PyDataclass
    block_list: Set[str] = Field(default_factory=set)

count: int = 4 class-attribute instance-attribute

post_dirname_format: str = '{title}' class-attribute instance-attribute

post_structure: PostStructureConfiguration = PostStructureConfiguration() class-attribute instance-attribute

mix_posts: bool = False class-attribute instance-attribute

sequential_filename: bool = False class-attribute instance-attribute

filename_format: str = '{}' class-attribute instance-attribute

allow_list: Set[str] = Field(default_factory=set) class-attribute instance-attribute

block_list: Set[str] = Field(default_factory=set) class-attribute instance-attribute

LoggerConfiguration

Logger configuration

Attributes:

Name	Type	Description
path	Optional[Path]	Path to save logs, None for disable log file output
level	Union[str, int]	Log filter level
rotation	Union[str, int, time, timedelta]	Log rotation

Source code in ktoolbox/configuration.py

class LoggerConfiguration(BaseModel):
    """
    Logger configuration

    :ivar path: Path to save logs, ``None`` for disable log file output
    :ivar level: Log filter level
    :ivar rotation: Log rotation
    """
    path: Optional[Path] = None
    level: Union[str, int] = logging.getLevelName(logging.DEBUG)
    rotation: Union[str, int, datetime.time, datetime.timedelta] = "1 week"

path: Optional[Path] = None class-attribute instance-attribute

level: Union[str, int] = logging.getLevelName(logging.DEBUG) class-attribute instance-attribute

rotation: Union[str, int, datetime.time, datetime.timedelta] = '1 week' class-attribute instance-attribute