Separate Token files based on data type

[aditya0by0]

This PR create separate tokens files for each type of data. So we could have separate tokens for chebi, and other molenet datasets.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 15 changed files in this pull request and generated no new comments.

Suppressed comments (6)

chebai_graph/preprocessing/properties/base.py:40

• Avoid using assert for required constructor arguments. assert can be stripped with Python optimizations (-O), which would let data_type=None through until it fails later. Raise a ValueError instead so callers always get a deterministic error.

        data_type: str,
        encoder: PropertyEncoder | None = None,
    ) -> None:
        assert data_type is not None, "data_type must be provided for MolecularProperty"
        if encoder is None:

chebai_graph/preprocessing/property_encoder.py:114

• The index cache loader currently includes empty/whitespace-only lines as valid tokens (e.g., the new indices files under preprocessing/bin/chebi/* end with a blank line). This adds an empty-string token to self.cache, which can change the class count and lead to incorrect encodings.

        self._data_type = data_type
        with open(self.index_path, "r") as pk:
            self.cache: dict[str, int] = {
                token.strip(): idx for idx, token in enumerate(pk)
            }

chebai_graph/preprocessing/properties/base.py:34

• The MolecularProperty docstring still documents only encoder, but data_type is now required to locate per-dataset token/index files. Updating the docstring makes the new API requirement discoverable for callers.

This issue also appears on line 36 of the same file.

        encoder: Optional encoder instance to encode property values.
                 Defaults to IndexEncoder if not provided.
    """

    def __init__(

chebai_graph/preprocessing/property_encoder.py:132

• Avoid using assert for runtime validation of data_type here; asserts can be disabled with Python optimizations (-O), which would allow invalid paths. Prefer raising a ValueError instead.

        assert self._data_type is not None, "data_type must be set for IndexEncoder"

chebai_graph/preprocessing/datasets/utils.py:25

• Docstring grammar: “This used to determine…” should be “This is used to determine…”.

        data_type (str): The data type associated with the property. This used to determine or set
            tokens file path for the property if applicable.

chebai_graph/preprocessing/datasets/utils.py:44

• resolve_property now always instantiates classes with data_type=.... That breaks resolution of custom/third-party MolecularProperty implementations whose __init__ doesn’t accept a data_type kwarg (previously, resolve_property supported arbitrary class paths). Consider a backwards-compatible fallback that retries without data_type only when the failure is due to an unexpected data_type kwarg.

        last_dot = property.rindex(".")
        module_name = property[:last_dot]
        class_name = property[last_dot + 1 :]
        module = importlib.import_module(module_name)
        return getattr(module, class_name)(data_type=data_type)

[aditya0by0]

I had trained a model before PR #41 , #42 , #43 #45 : https://wandb.ai/chebai/chebai/runs/l2v9td86/overview?nw=nwuseraditya0by0

val_macro-f1:
0.7307118773460388

val_micro-f1:
0.908219277858734

And Now I trained a model after the above PRs along with the changes in this PR with same hardware and parameters: https://wandb.ai/chebai/chebai/runs/qnxmoft7/overview?nw=nwuseraditya0by0

val_macro-f1:
0.7326366901397705

val_micro-f1:
0.9085060954093932

Ideally, the results should be identical down to last decimal, but there is a very minor difference, and this might be due to #45, hence I will conclude that the results are same.