Steve’s Data Tips and Tricks - Steve On Data

Categories

All (95)

apply (1)

arrow (1)

attributes (1)

augment (1)

autoarima (1)

automl (1)

benchmark (3)

bootstrap (2)

box (1)

cci30 (1)

code (88)

crypto (1)

cumulative (1)

data.table (2)

datatable (2)

distributions (1)

dplyr (1)

duckdb (1)

healthyr (7)

healthyrai (18)

healthyrdata (1)

healthyrts (15)

healthyverse (1)

histograms (2)

kmeans (2)

knn (1)

lapply (3)

linkedin (1)

list (1)

lists (5)

metadata (1)

mixturemodels (1)

modelr (1)

news (1)

parsnip (1)

plots (1)

preprocessor (1)

purrr (9)

randomwalk (2)

readr (1)

recipes (2)

rtip (83)

rvest (1)

sapply (1)

shell (1)

simulation (1)

skew (1)

sql (1)

thanks (1)

tidyaml (7)

tidydensity (14)

tidymodels (3)

timeseries (17)

transforms (1)

weeklytip (13)

workflowsets (1)

xgboost (2)

How fast does a compressed file in Part 2

rtip

benchmark

arrow

duckdb

datatable

readr

Yesterday I posted on performing a benchmark on reading in a compressed .csv.gz file of a 2,000 by 2,000 data.frame. It was brought to my attention by someone on Mastadon (@mariv…

Steven P. Sanderson II, MPH

How fast does a compressed file in?

rtip

benchmark

I received an email over the weekend in regards to my last post not containing the reading in of gz compressed file(s) for the benchmarking. While this was not an over site per-se it was a…

Steven P. Sanderson II, MPH

How fast do the files read in?

rtip

benchmark

I will demonstrate how to generate a 1,000 row and column matrix with random numbers in R, and then…

Steven P. Sanderson II, MPH

Some Examples of Cumulative Mean with {TidyDensity}

rtip

tidydensity

Cumulative mean is a statistical…

Steven P. Sanderson II, MPH

Getting the CCI30 Index Current Makeup

crypto

cci30

The CCI30 Crypto Index is a cryptocurrency index that tracks the…

Steven P. Sanderson II, MPH

Use of the apply family of functions

thanks

I am thrilled to announce that the R universe of packages {healthyverse} has surpassed 60,000…

Steven P. Sanderson II, MPH

Use of the apply family of functions

code

rtip

apply

In this post I will talk about the use of the R functions apply(), lapply(), sapply(), tapply(), and vapply() with examples.

Steven P. Sanderson II, MPH

Multiple Solutions to speedup tidy_bernoulli() with {data.table}

code

rtip

tidydensity

data.table

I had just recently posted on making an attempt to speedup computations with my package {TidyDensity} using a purely data.table solution, yes of course I can use {dtplyr} or {tidytabl…

Steven P. Sanderson II, MPH

Getting NYS Home Heating Oil Prices with {rvest}

code

rtip

rvest

If you live in New York and rely on heating oil to keep your home warm during the…

Steven P. Sanderson II, MPH

tidy_bernoulli() with {data.table}

code

rtip

data.table

So I was challanged by Adrian Antico to learn data.table, so yesterday I started with a single function from my package {TidyDensity} called tidy_bernoulli().

Steven P. Sanderson II, MPH

Simple examples of imap() from {purrr}

code

rtip

purrr

The imap() function is a powerful tool for iterating over a list or a vector while also keeping track of the index or names of…

Steven P. Sanderson II, MPH

Simple examples of pmap() from {purrr}

code

rtip

purrr

Steven P. Sanderson II, MPH

Forecasting Timeseries in a list with R

code

rtip

purrr

timeseries

In this article, we will discuss how to perform an ARIMA forecast on nested data or data that is in a list using R programming language. This is a common scenario in which…

Steven P. Sanderson II, MPH

Text Processing Made Easy with {healthyR}’s sql_left(), sql_mid(), and sql_right() Functions in R

code

rtip

sql

healthyr

Are you tired of manually manipulating…

Steven P. Sanderson II, MPH

Open a File Folder in R

code

rtip

shell

When writing a function, it is possible that…

Steven P. Sanderson II, MPH

Quickly Generate Nested Time Series Models

code

rtip

timeseries

autoarima

There are many approaches to modeling time series data in R. One of the types of data that we might come across is a nested time series. This means the data is grouped…

Steven P. Sanderson II, MPH

Data Preppers with {healthyR.ai}

code

rtip

healthyrai

preprocessor

There are many…

Steven P. Sanderson II, MPH

Calibrate and Plot a Time Series with {healthyR.ts}

code

rtip

healthyrts

timeseries

In time series analysis, it is common to split the data into training and testing sets to…

Steven P. Sanderson II, MPH

Converting a {tidyAML} tibble to a {workflowsets}

code

rtip

tidyaml

workflowsets

tidymodels

The {tidyAML} package…

Steven P. Sanderson II, MPH

Officially on CRAN {tidyAML}

code

rtip

tidyaml

I’m excited to announce that the R package {tidyAML} is now officially available on CRAN! This package is designed to make it easy for users to perform automated machine…

Steven P. Sanderson II, MPH

Moving Average Plots with {healthyR.ts}

code

rtip

timeseries

healthyrts

Are you interested in visualizing time series data in a clear and concise way? The R package {healthyR.ts} provides a…

Steven P. Sanderson II, MPH

An example of using {box}

code

rtip

box

Today I am going to make a short post on the R package {box} which was showcased to me quite nicely by Michael Miles. It was…

Steven P. Sanderson II, MPH

Off to CRAN! {tidyAML}

code

rtip

tidyaml

tidymodels

Are you tired of spending hours tuning and testing different machine learning models for your regression or classification problems? The new R package {tidyAML} is here to simplify the process for you! tidyAML is a simple interface for automatic machine learning that fits the tidymodels framework, making it easier for you to solve…

Steven P. Sanderson II, MPH

Get the Current Hospital Data Set from CMS with {healthyR.data}

code

rtip

healthyrdata

Getting data for health care in the US can sometimes be hard. With my R package {healthyR.data} I am hoping to alleviate some of that pain.

Steven P. Sanderson II, MPH

Creating and Predicting Fast Regression Parsnip Models with {tidyAML}

code

rtip

tidyaml

I am almost ready for a first release of my R package {tidyAML}. The purpose of this is to act as a way of quickly generating models using the parsnip package and…

Steven P. Sanderson II, MPH

Creating an R Project Directory

code

rtip

When working in R I find it best to create a new project when working on something. This keeps all of the data and scripts in one location. This also means that if you are…

Steven P. Sanderson II, MPH

Subsetting Named Lists in R

code

rtip

tidydensity

In R, lists are a fundamental data structure that allows us to store multiple objects of different data types under a single name. Often times, we want to extract certain…

Steven P. Sanderson II, MPH

Cumulative Measurement Functions with {TidyDensity}

code

rtip

tidydensity

If you’re looking for an easy-to-use package to calculate cumulative statistics in R, you may want to check out the TidyDensity package. This…

Steven P. Sanderson II, MPH

The Argument Matcher: A Function for Selecting the Right Arguments {tidyAML}

code

rtip

list

tidyaml

tidymodels

I am working on finishing up a few things with my new R package {tidyAML} before I release it to CRAN. One…

Steven P. Sanderson II, MPH

Diverging Lollipop Chart: A Visual Tool for Comparing Data with {healthyR}

code

rtip

healthyr

plots

A diverging lollipop chart is a useful tool for comparing data that falls into two categories, usually indicated by different colors.…

Steven P. Sanderson II, MPH

Attributes in R Functions: An Overview

code

rtip

metadata

attributes

R is a powerful programming language that is widely used for data analysis, visualization, and machine learning. One of the features of R that makes it versatile and…

Steven P. Sanderson II, MPH

Median: A Simple Way to Detect Excess Events Over Time with {healthyR}

code

rtip

timeseries

healthyr

As we collect data over time, it’s important to look for patterns and trends that can help us understand what’s happening. One common way to do this is to look at the median…

Steven P. Sanderson II, MPH

{healthyR.ts}: The New and Improved Library for Time Series Analysis

code

rtip

timeseries

healthyrts

Are you looking for a powerful and efficient library for time series analysis? Look no further than {healthyR.ts}! This library has recently been updated with new functions and improvements, making…

Steven P. Sanderson II, MPH

Service Line Grouping with {healthyR}

code

weeklytip

augment

healthyr

Healthcare data analysis can be a complex and time-consuming task, but it doesn’t have to be. Meet {healthyR}…

Steven P. Sanderson II, MPH

Transforming Your Data: A Guide to Popular Methods and How to Implement Them with {healthyR.ai}

code

rtip

transforms

healthyrai

Transforming data refers to the process of changing the…

Steven P. Sanderson II, MPH

Simplifying List Filtering in R with purrr’s keep()

code

rtip

lists

purrr

The {purrr} package in R is a powerful tool for working with lists and other data structures. One particularly useful function in the package is keep(), which allows you to filter a list by keeping only the elements…

Steven P. Sanderson II, MPH

Making Non Stationary Data Stationary

code

rtip

lists

timeseries

In the most basic sense for time series, a series is stationary if the properties of the generating process (the process that generates the data) do not change over time…

Steven P. Sanderson II, MPH

ADF and Phillips-Perron Tests for Stationarity using lists

code

rtip

lists

timeseries

lapply

A time series is a set of data points collected at regular intervals of time. Sometimes, the data points in a time series change…

Steven P. Sanderson II, MPH

Another Post on Lists

code

rtip

lists

Manipulating lists in R is a powerful tool for organizing and analyzing data. Here are a few common ways to manipulate lists:

Steven P. Sanderson II, MPH

Boilerplate XGBoost with {healthyR.ai}

code

rtip

xgboost

healthyrai

XGBoost, short for “eXtreme Gradient Boosting,” is a powerful…

Steven P. Sanderson II, MPH

Geometric Brownian Motion with {healthyR.ts}

code

rtip

timeseries

healthyrts

Geometric Brownian motion (GBM) is a widely used model in financial analysis for modeling the behavior of stock prices.…

Steven P. Sanderson II, MPH

Augmenting a Brownian Motion to a Time Series with {healthyR.ts}

code

rtip

timeseries

healthyrts

Time series analysis is a crucial tool for forecasting and understanding trends in various industries, including finance, economics, and…

Steven P. Sanderson II, MPH

Auto K-Means with {healthyR.ai}

code

rtip

kmeans

healthyrai

Today’s post is going to center around the automatic k-means functionality of {healthyR.ai}. I am not going to get into what it is or how it works, but rather…

Steven P. Sanderson II, MPH

The building of {tidyAML}

code

rtip

tidyaml

purrr

Yesterday I posted on An Update to {tidyAML} where I was discussing some of my thought process and how things could potentially work for the package.

Steven P. Sanderson II, MPH

An Update on {tidyAML}

code

rtip

tidyaml

automl

I have been doing a lot of work on a new package called {tidyAML}. {tidyAML} is a new R package that makes it easy to use the {tidymodels} ecosystem to perform automated machine learning (AutoML). This package provides a simple and intuitive interface that allows users to quickly generate machine learning…

Steven P. Sanderson II, MPH

Reflecting on the Past Year: A LinkedIn Year in Review

code

rtip

linkedin

Steven P. Sanderson II, MPH

Optimal Break Points for Histograms with {healthyR}

code

rtip

healthyr

histograms

Histogram binning is a technique used in data visualization to group continuous data into a set of discrete bins, or intervals. The purpose of histogram binning is to…

Steven P. Sanderson II, MPH

New Release of {healthyR.ts}

code

rtip

healthyrts

Steven P. Sanderson II, MPH

Brownian Motion

code

rtip

weeklytip

Brownian motion, also known as the random motion of particles suspended in a fluid, is a phenomenon that was…

Steven P. Sanderson II, MPH

More Randomwalks with {TidyDensity}

code

rtip

tidydensity

randomwalk

Random walks are a mathematical concept that have found various applications in fields such as economics, biology, and computer science. At a…

Steven P. Sanderson II, MPH

Calendar Heatmap with {healthyR.ts}

code

rtip

healthyrts

timeseries

Calendar heatmaps are a useful visualization tool for…

Steven P. Sanderson II, MPH

Event Analysis with {healthyR.ts}

code

rtip

healthyrts

timeseries

Time-to-event analysis, also known as survival analysis, is a statistical technique used to analyze the length of time until an event occurs. This type of analysis is often…

Steven P. Sanderson II, MPH

Gartner Magic Chart and its usefulness in healthcare analytics with {healthyR}

code

rtip

healthyr

The Gartner Magic Chart is a powerful tool for analyzing healthcare data and identifying trends and patterns that can inform decision making. It was…

Steven P. Sanderson II, MPH

Simulating Time Series Model Forecasts with {healthyR.ts}

code

weeklytip

healthyrts

timeseries

simulation

Time series models are a powerful tool for forecasting future values…

Steven P. Sanderson II, MPH

Listing Functions and Parameters

code

rtip

dplyr

I got a little bored one day and…

Steven P. Sanderson II, MPH

Distribution Statistics with {TidyDensity}

code

rtip

tidydensity

If you’re working with statistical distributions in R, you may be interested in the {TidyDensity} package. This package provides a set of functions for creating, manipulating, and…

Steven P. Sanderson II, MPH

Random Walks with {TidyDensity}

code

rtip

randomwalk

tidydensity

Random walks are a type of stochastic process that can be used to model the movement of a particle or system over time. At each time step, the position of the particle is…

Steven P. Sanderson II, MPH

Viewing Different Versions of the Same Statistical Distribution with {TidyDensity}

code

rtip

distributions

tidydensity

In statistics, it is often useful to view different versions of the same…

Steven P. Sanderson II, MPH

Model Scedacity Plots with {healthyR.ts}

code

rtip

timeseries

healthyrts

Scedacity plots are a useful tool for evaluating the performance of time series models and identifying trends or patterns in the data. They are a type of…

Steven P. Sanderson II, MPH

Simple Moving Average Plots with {healthyR.ts}

code

rtip

timeseries

healthyrts

The R package {healthyR.ts}, is an R package that allows users to easily plot and analyze their time series data. The package includes a variety of functions, but one of the standout features is the…

Steven P. Sanderson II, MPH

Distribution Summaries with {TidyDensity}

code

rtip

tidydensity

{TidyDensity} is an R package that provides tools for working with probability distributions in a tidy data format. One of the key functions in the package is tidy_distribut…

Steven P. Sanderson II, MPH

Mixture Distributions with {TidyDensity}

code

rtip

tidydensity

mixturemodels

A mixture distribution is a type of probability distribution that is created by combining two or more simpler distributions. This allows us to model…

Steven P. Sanderson II, MPH

Create QQ Plots for Time Series Models with {healthyR.ts}

code

rtip

timeseries

healthyrts

A Q-Q plot, or quantile-quantile plot, is a graphical tool for comparing two sets of data to assess whether they come from the same distribution. In the context of time…

Steven P. Sanderson II, MPH

Create a Faceted Historgram Plot with {healthyR.ai}

code

rtip

histograms

healthyrai

One of the most important steps in data analysis is visualizing the distribution…

Steven P. Sanderson II, MPH

Create Multiple {parsnip} Model Specs with {purrr}

code

rtip

parsnip

purrr

If you want to generate multiple parsnip model…

Steven P. Sanderson II, MPH

Z-Score Scaling Step Recipe with {healthyR.ai}

code

rtip

healthyrai

recipes

Sometimes one may find it useful or necessary to scale their data during a modeling or analysis phase. One of these such transformations is the z-score scaling.

Steven P. Sanderson II, MPH

Naming Items in a List with {purrr}, {dplyr}, or {healthyR}

code

rtip

purrr

healthyr

Many times when we are working with a data set we will want to break it up into groups and place them into a list and work with them…

Steven P. Sanderson II, MPH

Auto KNN with {healthyR.ai}

code

weeklytip

knn

healthyrai

Minimal coding ML is not something that is unheard of and is rather prolific, think h2o and pycaret just to name two. There is also no shortage available for R with the h2o interface…

Steven P. Sanderson II, MPH

Extract Boilerplate Workflow Metrics with {healthyR.ai}

code

rtip

healthyrai

When working with the {tidymodels} framework there are ways to pull model metrics from a workflow, since {healthyR.ai} is built on and around the {tidyverse} and {tidymodels} we can do the same. This post will focus on the…

Steven P. Sanderson II, MPH

Generate Random Walk Data with {healthyR.ts}

code

rtip

timeseries

healthyrts

Steven P. Sanderson II, MPH

Working with Lists

code

rtip

lists

lapply

In R there are many times where we will work with lists. I won’t go into why lists are great or really the structure of a list but rather simply working with them.

Steven P. Sanderson II, MPH

Default Metric Sets with {healthyR.ai}

code

rtip

healthyrai

When modeling it is always good to understand your model performance against some metric The {tidymodels} package {yardstick} is a great resource for this.

Steven P. Sanderson II, MPH

Summary Statistics with {TidyDensity}

code

weeklytip

tidydensity

datatable

Many times someone may want to see a summary or cumulative statistic for a given set of data or even from several simulations of data. I went over bootstrap plotting earlier this month, and this is a form of what we will go over today although slightly more…

Steven P. Sanderson II, MPH

Data Preprocessing Scale/Normalize with {healthyR.ai}

code

rtip

healthyrai

recipes

A large portion of data modeling occurrs not only in the data cleaning phase but also in the data preprocessing phase. This can include things like scaling or normalizing data before proceeding to the modeling phase. I will discuss one such function…

Steven P. Sanderson II, MPH

Bootstrap Modeling with Base R

code

bootstrap

rtip

I have previously written about bootstrap modeling with {purrr} and {modelr} here. What if you would like to do some simple bootstrap modeling without importing a…

Steven P. Sanderson II, MPH

Updates to {healthyverse} packages

code

weeklytip

healthyverse

I have made several updates to {healthyverse}, this has resulted in new releases to CRAN for {healthyR.ai}, {healthyR.ts}, and {TidyDesnsity}.

Steven P. Sanderson II, MPH

Bootstrap Modeling with {purrr} and {modler}

code

rtip

purrr

modelr

Many times in modeling we want to get the uncertainty in the model, well, bootstrapping to the rescue!

Steven P. Sanderson II, MPH

Cumulative Harmonic Mean with {TidyDensity}

code

rtip

tidydensity

There can be times in which you may want to see a cumulative statistic, maybe in this particular case it is the harmonic mean. Well with the {TidyDensity} it is possible…

Steven P. Sanderson II, MPH

Auto Prep data for XGBoost with {healthyR.ai}

code

rtip

healthyrai

xgboost

Sometimes we may want to…

Steven P. Sanderson II, MPH

Find Skewed Features with {healthyR.ai}

code

rtip

healthyrai

skew

Sometimes we may want to quickly find skewed features in a data set. This is easily achiveable using the {healthyR.ai} library. There is a simple function called hai_ske…

Steven P. Sanderson II, MPH

Time Series Lag Correlation Plots

code

rtip

healthyrts

weeklytip

In time series analysis there is something called a lag. This simply means we take a look at some past event from some point in time t. This is a non-statistical method for looking at a…

Steven P. Sanderson II, MPH

Reading Multiple Files with {purrr}

code

rtip

purrr

There may be times when you have multiple structured files in the same folder, maybe they are .csv files. For this short tip, we will say that they are.

Steven P. Sanderson II, MPH

Mapping K-Means with healthyR.ai

code

rtip

healthyrai

kmeans

K-Means is a clustering algorithm that can be used to find potential clusters in your data.

Steven P. Sanderson II, MPH

Hyperbolic Transform with healthyR.ai

code

rtip

healthyrai

In data modeling there can be instanes where you will want some sort of hyperbolic transformation of your data. In {healthyR.ai} this is easy with the use of the function hai_hyperbolic_…

Steven P. Sanderson II, MPH

Discrete Fourier Vec with healthyR.ai

code

rtip

healthyrai

Sometimes in modeling you may want to get a discrete 1/0 vector of a fourier transform of some input vector. With {healthyR.ai} we can do this easily.

Steven P. Sanderson II, MPH

Bootstrapping and Plots with TidyDensity

code

tidydensity

bootstrap

weeklytip

Many times in the real world we have a data set which is actually a sample as we typically do not know what the actual population is. This is where bootstrapping tends to come into play. It allows us to get a hold on what the possible parameter values are by taking repeated samples of the data that is…

Steven P. Sanderson II, MPH

Cumulative Skewness

code

rtip

In this post we will make a function cum_skewness() that will generate a vector output of the cumulative skewness of some given vector. The full function call is simply:

Steven P. Sanderson II, MPH

PCA with healthyR.ai

code

rtip

healthyrai

weeklytip

In this post we are going to talk about how you can perform principal component analysis in R with {healthyR.ai} in a tidyverse compliant fashion.

Steven P. Sanderson II, MPH

Control Charts in healthyR.ai

code

rtip

healthyrai

Sometimes you may be working with a time series or some process data and you will want to make a control chart. This is simple to do with the {healthyR.ai} package.

Steven P. Sanderson II, MPH

Cumulative Variance

code

rtip

cumulative

sapply

lapply

This is going to be a simple example on how we can make a function in #base #r that will crate a cumulative variance function. From base R we are going to use seq_along(), stats::v…

Steven P. Sanderson II, MPH

Time Series Clustering with healthyR.ts

code

weeklytip

healthyrts

There are two components to time-series clustering with {healthyR.ts}. There is the function that will create the clustering data along with a slew of other information…

Steven P. Sanderson II, MPH

healthyR.ai Primer

code

weeklytip

healthyrai

This is going to serve as a sort of primer for my r packge {healthyR.ai}. The goal of this package is to help with producing uniform machine learning/ai models either from scratch…

Steven P. Sanderson II, MPH

TidyDensity Primer

code

weeklytip

tidydensity

This is going to serve as a sort of primer for the {TidyDensity} package.

Steven P. Sanderson II, MPH

Simple lapply()

code

rtip

weeklytip

This is a simple lapply example to start things off.

Steven P. Sanderson II, MPH

Welcome To Steve On Data

news

This is the first post in a Quarto blog. Welcome!

Steven P. Sanderson II, MPH