Skip to main content

32 posts tagged with "pandas"

pandas, the Python Library for Tables of Data and DataFrames

View All Tags

How to Load a Logged Table from MLflow with load_table

· 5 min read
Jagdish Kumawat
Founder @ Dewiride

A table logged with mlflow.log_table() can be read straight back into a pandas DataFrame with mlflow.load_table(). Its best trick is reading the same table from many runs at once, stacked into one DataFrame with a column that says which run each row came from. This guide loads a predictions table from two runs, with real output from MLflow 3.18.0.

How to Log a Table in MLflow with log_table

· 6 min read
Jagdish Kumawat
Founder @ Dewiride

Some results of a run are tables: predictions next to the real values, scores per group, the importance of each feature. mlflow.log_table() saves a pandas DataFrame or a dictionary with the run as a table that the MLflow UI shows as rows and columns and can compare across runs. This guide logs a small predictions table from two models, with real output from MLflow 3.18.0.

How to Make an MLflow Dataset Digest Cover Every Row

· 8 min read
Jagdish Kumawat
Founder @ Dewiride

An MLflow dataset's digest is meant to tell you when data changes, but for a pandas DataFrame it only looks at part of it: the first 10,000 rows, and only the columns that hold text or numbers. Change a value further down, or a date anywhere, and the digest stays the same. This guide shows both gaps happening, finds the reason in MLflow's code, and passes a digest of every row and column instead, with real output from MLflow 3.18.0.

How to Log a Dataset in MLflow

· 8 min read
Jagdish Kumawat
Founder @ Dewiride

A model is only as good as the data it learned from, so a run should record which data that was. MLflow does it in two calls: mlflow.data.from_pandas() describes a pandas DataFrame as a dataset, and mlflow.log_input() attaches that description to the run. This guide logs a small CSV file of houses, with real output from MLflow 3.18.0.

How to Save a pandas DataFrame to a CSV File

· 4 min read
Jagdish Kumawat
Founder @ Dewiride

A CSV file is the simplest way to hand a table to someone else: Excel, Google Sheets and nearly every tool can open it. DataFrame.to_csv writes one in a single call, but its default adds the table's row numbers as an extra, nameless first column. This guide writes the same table twice, with and without them, and shows both files, with real output from pandas 3.1.1.

How to Sort a pandas DataFrame by a Column

· 5 min read
Jagdish Kumawat
Founder @ Dewiride

Sorting a table by one column is how you find its top or bottom rows: the most expensive houses, or the predictions a model got most wrong. DataFrame.sort_values does it in one call, and two of its options cover cases that trip people up: sorting by size regardless of sign, and renumbering the rows afterwards. This guide sorts a small table of prediction errors, with real output from pandas 3.1.1.

How to Join a pandas Series to a DataFrame by Index

· 5 min read
Jagdish Kumawat
Founder @ Dewiride

When a table's columns and one more column live apart, such as the inputs of a model and the value it should predict, DataFrame.join puts them back together. It matches rows by their labels, not their position, so the right value lands on the right row even when the two are in a different order. This guide joins a price column to a table of houses, with real output from pandas 3.1.1.

How to Create a pandas DataFrame from a Dictionary

· 5 min read
Jagdish Kumawat
Founder @ Dewiride

A dictionary of lists is the most direct way to type a small table in Python: each key is a column's name and each list holds that column's values. pd.DataFrame() turns it into a pandas table and works out a type for every column. This guide builds a three-row table of houses, with real output from pandas 3.1.1.

Fixed: UserWarning "Hint: Inferred schema contains integer column(s)" When You Log a Dataset in MLflow

· 9 min read
Jagdish Kumawat
Founder @ Dewiride

Logging a small table of whole numbers as an MLflow dataset printed a long UserWarning: Hint: Inferred schema contains integer column(s). Integer columns in Python cannot represent missing values. The hint is about models, whose input types MLflow checks at prediction time, and nothing checks a dataset's schema. Whether you silence it or switch to decimal columns depends on one question: can those columns ever be empty?

Fixed: UserWarning "The specified dataset source can be interpreted in multiple ways: LocalArtifactDatasetSource, LocalArtifactDatasetSource" in MLflow

· 10 min read
Jagdish Kumawat
Founder @ Dewiride

Logging a pandas DataFrame read from a CSV file, with that file as its source, made MLflow print UserWarning: The specified dataset source can be interpreted in multiple ways: LocalArtifactDatasetSource, LocalArtifactDatasetSource. It names the same source type twice because MLflow 3.18.0 registers it twice. The dataset it logs is correct, and a warnings filter for this one message keeps your output clean until MLflow fixes it.