Introduction

Working with R dataframes and matrices often means manipulating column names for data processing, exporting, or performance. Whether you need to remove headers for compatibility or simply want a cleaner structure, R offers several ways to remove column names. In this guide, you’ll learn how to use unname(), colnames(), and related functions, with clear examples and practical advice.

Understanding Column Names in R

Column names (sometimes called headers) are labels for each column in a dataframe or matrix. They help identify data, but sometimes you need to remove them—for example, before exporting data to a system that doesn’t expect headers, or for certain performance optimizations.

Method 1: Using unname() Function

unname(object, force = FALSE)

# Create a matrix with column names
m <- matrix(1:6, nrow = 2)
colnames(m) <- c("A", "B", "C")
print(m)

     A B C
[1,] 1 3 5
[2,] 2 4 6

# Remove column names using unname()
m2 <- unname(m)
print(m2)

     [,1] [,2] [,3]
[1,]    1    3    5
[2,]    2    4    6

colnames(m2) # Returns NULL

NULL

# Create a matrix with row and column names
m <- matrix(1:4, nrow = 2)
rownames(m) <- c("row1", "row2")
colnames(m) <- c("col1", "col2")

# View the matrix with names
print(m)

     col1 col2
row1    1    3
row2    2    4

dimnames(m)

[[1]]
[1] "row1" "row2"

[[2]]
[1] "col1" "col2"

# Remove all names using unname() with force = TRUE
m2 <- unname(m, force = TRUE)

# View the result
print(m2)

     [,1] [,2]
[1,]    1    3
[2,]    2    4

dimnames(m2)

NULL

Method 2: Setting colnames() to NULL

m <- matrix(1:6, nrow = 2)

colnames(m) <- c("A", "B", "C")
cat("The column names are: ", colnames(m), "\n")

The column names are:  A B C 

colnames(m) <- NULL
print(m)

     [,1] [,2] [,3]
[1,]    1    3    5
[2,]    2    4    6

cat("\nThe column names are:", colnames(m), "\n") # Returns NULL

The column names are: 

df <- data.frame(a = 1:3, b = 4:6)
colnames(df) <- NULL # Error: invalid 'names' attribute
colnames(df)

NULL

names(df)

NULL

dimnames(df)

[[1]]
[1] "1" "2" "3"

[[2]]
NULL

Method 3: Using names() and setNames()

v <- c(a = 1, b = 2, c = 3)
names(v) <- NULL
print(v)

[1] 1 2 3

# [1] 1 2 3

# Or using setNames()
v2 <- setNames(v, NULL)
print(v2)

[1] 1 2 3

# [1] 1 2 3

Removing Column Names from Data Frames: The Safe Way

Since R data frames must have column names, the safest way to “remove” them is to convert the data frame to a matrix:

df <- data.frame(a = 1:3, b = 4:6)
m <- as.matrix(df)
colnames(m) <- NULL
print(m)

     [,1] [,2]
[1,]    1    4
[2,]    2    5
[3,]    3    6

This approach is widely used in practice .

Comparison Table: Methods for Removing Column Names

Your Turn!

Task:
Given the following matrix, remove its column names and print the result.

m <- matrix(1:9, nrow = 3)
colnames(m) <- c("X", "Y", "Z")

# Remove column names
colnames(m) <- NULL
print(m)

     [,1] [,2] [,3]
[1,]    1    4    7
[2,]    2    5    8
[3,]    3    6    9

Key Takeaways

• ✅ Use colnames(m) <- NULL or unname(m) to remove column names from matrices.
• ✅ Data frames in R require column names; to remove them, convert to a matrix first.
• ✅ For vectors and lists, use names(x) <- NULL or unname(x).
• ✅ The unname() function with force = TRUE can remove names from data frames, but may not be supported in all R versions.
• ✅ Always check your object structure after removing names, especially for data frames.

Conclusion

Removing column names in R is straightforward for matrices and vectors, but requires care with data frames. Use colnames() or unname() for matrices, and always convert data frames to matrices if you need to strip headers. These techniques help you prepare data for export, improve performance, or meet specific analysis requirements.

Ready to streamline your R data workflows? Try these methods in your next project!

Frequently Asked Questions

1. Can I remove column names from a data frame in R?
No, R data frames require valid column names. To remove them, convert the data frame to a matrix first.

2. What is the difference between unname() and colnames() <- NULL?
Both remove column names from matrices, but unname() can also remove names from vectors and lists.

3. Will removing column names affect my data?
No, the data remains unchanged, but you lose the labels that help identify columns.

4. Can I remove row names the same way?
Yes, use rownames(m) <- NULL for matrices, or unname() for vectors.

5. Is it safe to use unname(df, force = TRUE) on data frames?
It may work in some R versions, but can cause errors. The safest approach is to convert to a matrix first.

References

1. R Documentation: unname() function
2. R Documentation: colnames() function
3. R Documentation: names() function

Enjoyed this guide?
Share your feedback below and connect with fellow R programmers!
If you found this helpful, share

Happy Coding! 🚀

You can connect with me at any one of the below:

Telegram Channel here: https://t.me/steveondata

LinkedIn Network here: https://www.linkedin.com/in/spsanderson/

Mastadon Social here: https://mstdn.social/@stevensanderson

RStats Network here: https://rstats.me/@spsanderson

GitHub Network here: https://github.com/spsanderson

Bluesky Network here: https://bsky.app/profile/spsanderson.com

My Book: Extending Excel with Python and R here: https://packt.link/oTyZJ

You.com Referral Link: https://you.com/join/EHSLDTL6

@online{
  how_to_remove_column_names_in_r_complete_guide_to_unname_and_alternative_methods_20251201,
  author = {Sanderson II MPH, Steven P.},
  title = {How to Remove Column Names in R: Complete Guide to unname() and Alternative Methods},
  date = {2025-12-01},
  url = {https://www.spsanderson.com/steveondata/posts/2025-12-01/},
  langid = {en}
  }

Introduction

For-loops are a cornerstone of R programming, allowing you to repeat actions for each element in a sequence or range. Whether you’re iterating over numbers, vectors, or data frames, understanding how to use ranges in for-loops is essential for data analysis, automation, and algorithm development.

This article will walk you through the essentials of for-loops with ranges in R, from simple counting to advanced applications, with clear, working code examples and practical tips.

What is a For-Loop in R?

A for-loop in R is a control structure that repeats a block of code for each value in a sequence (range). It’s especially useful when you need to perform repetitive tasks, process data row by row, or apply custom logic that isn’t easily vectorized .

Basic For-Loop Syntax

The general syntax is:

for (variable in sequence) {
  # Code to execute
}

• variable takes each value from sequence in turn.
• The code inside the braces {} runs for each value.

Creating Ranges: Colon Operator and seq()

1:5   # Creates the sequence 1, 2, 3, 4, 5
10:1  # Creates the sequence 10, 9, 8, ..., 1

seq(from = 0, to = 10, by = 2)    # 0, 2, 4, 6, 8, 10

[1]  0  2  4  6  8 10

seq(0, 1, by = 0.1)               # 0.0, 0.1, ..., 1.0

 [1] 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

seq_along(vector)                 # Indices for a vector

[1] 1

Working Examples: From Simple to Advanced

# Count from 1 to 10
for (i in 1:10) {
  print(paste("Count:", i))
}

[1] "Count: 1"
[1] "Count: 2"
[1] "Count: 3"
[1] "Count: 4"
[1] "Count: 5"
[1] "Count: 6"
[1] "Count: 7"
[1] "Count: 8"
[1] "Count: 9"
[1] "Count: 10"

# Countdown from 10 to 1
for (i in 10:1) {
  print(paste("Countdown:", i))
}

[1] "Countdown: 10"
[1] "Countdown: 9"
[1] "Countdown: 8"
[1] "Countdown: 7"
[1] "Countdown: 6"
[1] "Countdown: 5"
[1] "Countdown: 4"
[1] "Countdown: 3"
[1] "Countdown: 2"
[1] "Countdown: 1"

# Even numbers from 0 to 20
for (i in seq(0, 20, by = 2)) {
  print(paste("Even number:", i))
}

[1] "Even number: 0"
[1] "Even number: 2"
[1] "Even number: 4"
[1] "Even number: 6"
[1] "Even number: 8"
[1] "Even number: 10"
[1] "Even number: 12"
[1] "Even number: 14"
[1] "Even number: 16"
[1] "Even number: 18"
[1] "Even number: 20"

# Sine values for decimals between 0 and 1
for (x in seq(0, 1, by = 0.1)) {
  print(paste("sin(", round(x, 1), ") =", round(sin(x), 3)))
}

[1] "sin( 0 ) = 0"
[1] "sin( 0.1 ) = 0.1"
[1] "sin( 0.2 ) = 0.199"
[1] "sin( 0.3 ) = 0.296"
[1] "sin( 0.4 ) = 0.389"
[1] "sin( 0.5 ) = 0.479"
[1] "sin( 0.6 ) = 0.565"
[1] "sin( 0.7 ) = 0.644"
[1] "sin( 0.8 ) = 0.717"
[1] "sin( 0.9 ) = 0.783"
[1] "sin( 1 ) = 0.841"

fruits <- c("apple", "banana", "cherry")
for (fruit in fruits) {
  print(paste("I like", fruit))
}

[1] "I like apple"
[1] "I like banana"
[1] "I like cherry"

scores <- c(85, 92, 78)
for (i in seq_along(scores)) {
  print(paste("Score", i, "is", scores[i]))
}

[1] "Score 1 is 85"
[1] "Score 2 is 92"
[1] "Score 3 is 78"

# Multiplication table (1-3)
for (i in 1:3) {
  for (j in 1:3) {
    cat(sprintf("%d x %d = %d\n", i, j, i*j))
  }
}

1 x 1 = 1
1 x 2 = 2
1 x 3 = 3
2 x 1 = 2
2 x 2 = 4
2 x 3 = 6
3 x 1 = 3
3 x 2 = 6
3 x 3 = 9

employees <- data.frame(
  name = c("Alice", "Bob"),
  age = c(25, 30)
)
for (i in 1:nrow(employees)) {
  print(paste(employees$name[i], "is", employees$age[i], "years old"))
}

[1] "Alice is 25 years old"
[1] "Bob is 30 years old"

temps <- c(22, 18, 25, 30)
hot_days <- c()
for (i in seq_along(temps)) {
  if (temps[i] >= 25) {
    hot_days <- c(hot_days, i)
  }
}
print(paste("Hot days at positions:", paste(hot_days, collapse = ", ")))

[1] "Hot days at positions: 3, 4"

n <- 10
squares <- numeric(n)
for (i in 1:n) {
  squares[i] <- i^2
}
print(squares)

 [1]   1   4   9  16  25  36  49  64  81 100

numbers <- c(1, 2, -1, 3, 0, 4)
sum_pos <- 0
for (num in numbers) {
  if (num < 0) next
  if (num == 0) break
  sum_pos <- sum_pos + num
}
print(sum_pos)

[1] 6

Best Practices and Common Pitfalls

Your Turn!

Write a for-loop that calculates the running average of a numeric vector.

# Given vector
scores <- c(85, 92, 78, 96, 89, 94, 87, 91)
# Your code here

running_avg <- numeric(length(scores))
for (i in 1:length(scores)) {
  running_avg[i] <- mean(scores[1:i])
  print(paste("Position", i, "- Running Average:", round(running_avg[i], 2)))
}

[1] "Position 1 - Running Average: 85"
[1] "Position 2 - Running Average: 88.5"
[1] "Position 3 - Running Average: 85"
[1] "Position 4 - Running Average: 87.75"
[1] "Position 5 - Running Average: 88"
[1] "Position 6 - Running Average: 89"
[1] "Position 7 - Running Average: 88.71"
[1] "Position 8 - Running Average: 89"

Key Takeaways

• Use : for simple integer ranges, seq() for custom steps or decimals.
• Prefer seq_along() over 1:length() for safe iteration.
• Pre-allocate vectors for better performance.
• Use break and next for flexible loop control.
• For-loops are great for custom logic; use vectorized functions when possible.

FAQs

Q1: When should I use : vs seq()?
A1: Use : for simple, consecutive integers. Use seq() for custom steps, decimals, or non-integer sequences.

Q2: Can I change the loop variable inside the loop?
A2: You can, but it won’t affect the sequence of values the loop iterates over.

Q3: What’s the difference between for-loops and apply functions?
A3: For-loops offer explicit control and are easier to debug; apply functions are concise and often faster for simple operations.

Q4: How do I avoid errors with empty vectors?
A4: Use seq_along() or seq_len() instead of 1:length() to handle empty vectors safely.

Q5: What happens if I use 1:0 in a for-loop?
A5: It creates the sequence c(1, 0), so the loop runs twice. Use seq_len() for zero-length safety.

Conclusion

For-loops with ranges are a powerful tool in R, enabling you to automate repetitive tasks, process data, and implement custom logic. By mastering range creation, safe iteration, and best practices, you’ll write more robust and efficient R code. Practice with the examples above and integrate these techniques into your projects for better results.

References

1. R Core Team. (2024). An Introduction to R. CRAN.
2. Wickham, H., & Grolemund, G. (2023). R for Data Science (2nd Edition).
3. Data Carpentry. (2022). For Loops in R.
4. GeeksforGeeks. (2020). For loop in R.

Enjoyed this guide?
Share it with your fellow R programmers and bookmark it for future reference!

Happy Coding! 🚀

You can connect with me at any one of the below:

Telegram Channel here: https://t.me/steveondata

LinkedIn Network here: https://www.linkedin.com/in/spsanderson/

Mastadon Social here: https://mstdn.social/@stevensanderson

RStats Network here: https://rstats.me/@spsanderson

GitHub Network here: https://github.com/spsanderson

Bluesky Network here: https://bsky.app/profile/spsanderson.com

My Book: Extending Excel with Python and R here: https://packt.link/oTyZJ

You.com Referral Link: https://you.com/join/EHSLDTL6

@online{
  for_loop_with_range_in_r_a_complete_guide_with_practical_examples_20251117,
  author = {Sanderson II MPH, Steven P.},
  title = {For-Loop with Range in R: A Complete Guide with Practical Examples},
  date = {2025-11-17},
  url = {https://www.spsanderson.com/steveondata/posts/2025-11-17/},
  langid = {en}
  }

Why Use Python for Email and SMS?

Python is one of the most popular programming languages for automating tasks—including sending emails and text messages. With just a few lines of code, you can:

• Send notifications automatically
• Alert yourself about important events
• Build contact forms or customer support tools
• Save time by automating repetitive communication

What You Need to Get Started

• Python 3.x installed on your computer
• A code editor (like VS Code, PyCharm, or even Notepad++)
• Internet access
• For emails: a Gmail (or other SMTP) account
• For SMS: a Twilio account (free trial available)

Required Libraries: - smtplib (built-in for emails) - email (built-in for formatting emails) - twilio (install with pip install twilio for SMS)

Sending Emails with Python (Step-by-Step)

import smtplib
from email.message import EmailMessage

# Set up your email details
EMAIL_ADDRESS = 'your_email@gmail.com'
EMAIL_PASSWORD = 'your_app_password'

msg = EmailMessage()
msg['Subject'] = 'Hello from Python!'
msg['From'] = EMAIL_ADDRESS
msg['To'] = 'recipient@example.com'
msg.set_content('This is a test email sent from Python!')

# Connect to Gmail SMTP server and send email
with smtplib.SMTP('smtp.gmail.com', 587) as smtp:
    smtp.starttls()  # Secure the connection
    smtp.login(EMAIL_ADDRESS, EMAIL_PASSWORD)
    smtp.send_message(msg)

print("Email sent successfully!")

import smtplib
from email.message import EmailMessage

EMAIL_ADDRESS = 'your_email@gmail.com'
EMAIL_PASSWORD = 'your_app_password'

msg = EmailMessage()
msg['Subject'] = 'HTML Email with Attachment'
msg['From'] = EMAIL_ADDRESS
msg['To'] = 'recipient@example.com'
msg.set_content('This is a plain text fallback.')

# Add HTML content
msg.add_alternative("""
<html>
  <body>
    <h1 style="color:blue;">Hello from Python!</h1>
    <p>This is an <b>HTML email</b> with an attachment.</p>
  </body>
</html>
""", subtype='html')

# Attach a file
with open('example.pdf', 'rb') as f:
    file_data = f.read()
    file_name = f.name
msg.add_attachment(file_data, maintype='application', subtype='pdf', filename=file_name)

with smtplib.SMTP('smtp.gmail.com', 587) as smtp:
    smtp.starttls()
    smtp.login(EMAIL_ADDRESS, EMAIL_PASSWORD)
    smtp.send_message(msg)

print("HTML email with attachment sent!")

import re

def is_valid_email(email):
    pattern = r'^[\w\.-]+@[\w\.-]+\.\w+$'
    return re.match(pattern, email) is not None

print(is_valid_email('test@example.com'))  # True
print(is_valid_email('not-an-email'))      # False

Sending SMS with Python (Twilio)

pip install twilio

from twilio.rest import Client

# Your Twilio credentials
account_sid = 'your_account_sid'
auth_token = 'your_auth_token'
client = Client(account_sid, auth_token)

message = client.messages.create(
    body='Hello from Python SMS!',
    from_='+1234567890',  # Your Twilio number
    to='+0987654321'      # Recipient's phone number
)

print("SMS sent! Message SID:", message.sid)

Error Handling & Troubleshooting

• SMTP Connection Refused: Double-check your SMTP server, port, and internet connection. Make sure your firewall isn’t blocking port 587.
• Authentication Errors: Use App Passwords for Gmail. For Twilio, make sure your Account SID and Auth Token are correct.
• SMS Not Sending: Check if your Twilio trial number is verified and you have enough credits.

Security Best Practices

• Never hard-code passwords or API keys in your scripts. Use environment variables or a .env file.
• Use App Passwords for Gmail, not your main password.
• Keep your credentials private—never share them or upload them to public repositories.

Practical Examples & Use Cases

• Automated Email Alerts: Get notified when a script finishes or an error occurs.
• Order Confirmations: Send customers a confirmation email or SMS after a purchase.
• Contact Forms: Automatically email or text yourself when someone fills out a form on your website.

Your Turn!

Try sending an email and an SMS using the code above. As a challenge, modify the email script to send a message to multiple recipients at once.

# Sending to multiple recipients
msg['To'] = ', '.join(['person1@example.com', 'person2@example.com'])

Quick Takeaways

• Python makes sending emails and SMS easy with smtplib and Twilio.
• Always use secure methods for storing credentials.
• Validate email addresses before sending.
• Automate notifications to save time and reduce manual work.
• Troubleshoot common errors by checking settings and credentials.

Conclusion & Next Steps

You’ve just learned how to send emails and text messages with Python! With these skills, you can automate notifications, build smarter apps, and save tons of time. Keep experimenting, try new features, and don’t hesitate to ask for help or share your progress.

Ready to level up? Try building a simple notification system for your next project, or explore more advanced email and SMS features!

FAQs

1. Do I need to pay to send emails with Python?
No! Sending emails with Python’s smtplib is free. SMS via Twilio is free with trial credits, but paid after that.

2. Is it safe to use my Gmail password in Python scripts?
Never use your regular password. Always use an App Password for scripts.

3. Why do I get ‘connection refused’ errors?
Check your SMTP server, port, and firewall settings. For Gmail, use smtp.gmail.com:587 with TLS.

4. Can I send SMS without Twilio?
Yes! Alternatives include TextBelt, MessageBird, and Nexmo.

5. How many emails can I send per day with Python?
Gmail allows 500/day for regular accounts. Other providers have different limits.

Engage!

Did this guide help you?
Share your experience, ask questions, or suggest improvements in the comments below! If you found this useful, share it with friends or on social media to help more beginners discover Python automation.

You’re now ready to automate your communications with Python. Happy coding!

References

• How to Send Emails Using Python and Gmail SMTP Server
• Gmail SMTP Server Settings for Sending Emails

• Python smtplib — SMTP protocol client

Happy Coding! 🚀

You can connect with me at any one of the below:

Telegram Channel here: https://t.me/steveondata

LinkedIn Network here: https://www.linkedin.com/in/spsanderson/

Mastadon Social here: https://mstdn.social/@stevensanderson

RStats Network here: https://rstats.me/@spsanderson

GitHub Network here: https://github.com/spsanderson

Bluesky Network here: https://bsky.app/profile/spsanderson.com

My Book: Extending Excel with Python and R here: https://packt.link/oTyZJ

You.com Referral Link: https://you.com/join/EHSLDTL6

@online{
    sending_email_and_text_messages_with_python_a_beginner_s_complete_guide_20251106,
    author = {Sanderson II MPH, Steven P.},
    title = {Sending Email and Text Messages with Python: A Beginner’s Complete Guide},
    date = {2025-11-06}, url = {https://www.spsanderson.com/steveondata/posts/2025-11-06/},
    langid = {en} 
    }

Introduction

Nested for loops are a fundamental tool in R programming, especially when working with multi-dimensional data like matrices or performing repetitive tasks across rows and columns. While R offers many ways to iterate, understanding how to use nested for loops effectively is essential for every R programmer. In this article, you’ll learn the syntax, see real-world examples, discover best practices, and avoid common mistakes when using nested for loops in R.

What is a Nested For Loop?

A nested for loop is simply a for loop inside another for loop. This structure allows you to iterate over two (or more) dimensions—think of looping through every cell in a table, or comparing every element in one vector to every element in another. Nested loops are especially useful for matrix operations, pairwise comparisons, and complex data transformations .

Understanding the Basic Syntax

The general syntax for a nested for loop in R is:

for (i in seq1) {
  for (j in seq2) {
    # Code to execute
  }
}

• i and j are loop variables.
• seq1 and seq2 are sequences (like 1:5, letters, or any vector).

Step-by-Step Examples

for (i in 1:3) {
  for (j in 1:2) {
    print(paste("Row:", i, "Col:", j))
  }
}

[1] "Row: 1 Col: 1"
[1] "Row: 1 Col: 2"
[1] "Row: 2 Col: 1"
[1] "Row: 2 Col: 2"
[1] "Row: 3 Col: 1"
[1] "Row: 3 Col: 2"

n <- 3
m <- 4
result <- matrix(0, n, m)
for (i in 1:n) {
  for (j in 1:m) {
    result[i, j] <- i * j
  }
}
print(result)

     [,1] [,2] [,3] [,4]
[1,]    1    2    3    4
[2,]    2    4    6    8
[3,]    3    6    9   12

df <- data.frame(A = 1:3, B = 4:6)
for (i in 1:nrow(df)) {
  for (j in 1:ncol(df)) {
    print(paste("Row", i, "Col", j, "Value:", df[i, j]))
  }
}

[1] "Row 1 Col 1 Value: 1"
[1] "Row 1 Col 2 Value: 4"
[1] "Row 2 Col 1 Value: 2"
[1] "Row 2 Col 2 Value: 5"
[1] "Row 3 Col 1 Value: 3"
[1] "Row 3 Col 2 Value: 6"

Real-World Use Cases

Control Flow in Nested Loops

for (i in 1:3) {
  for (j in 1:3) {
    if (i == j) next
    print(paste(i, j))
  }
}

[1] "1 2"
[1] "1 3"
[1] "2 1"
[1] "2 3"
[1] "3 1"
[1] "3 2"

Performance Considerations and Optimization

# Using apply for matrix operations
result <- outer(1:3, 1:4, FUN = "*")

library(foreach)
library(doParallel)
registerDoParallel(cores = 2)
result <- foreach(i = 1:3, .combine = rbind) %:%
            foreach(j = 1:4, .combine = c) %dopar% {
              i * j
            }
print(result)

         [,1] [,2] [,3] [,4]
result.1    1    2    3    4
result.2    2    4    6    8
result.3    3    6    9   12

stopImplicitCluster()

Common Mistakes and Debugging Tips

• Growing objects inside loops: Avoid using c() or rbind() repeatedly inside loops; preallocate instead.
• Index out of bounds: Always check your loop ranges to avoid errors.
• Misusing break/next: Remember, they only affect the innermost loop.

n <- 5
result <- matrix(0, n, n)
for (i in 1:n) {
  for (j in 1:n) {
    result[i, j] <- i + j
  }
}
print(result)

     [,1] [,2] [,3] [,4] [,5]
[1,]    2    3    4    5    6
[2,]    3    4    5    6    7
[3,]    4    5    6    7    8
[4,]    5    6    7    8    9
[5,]    6    7    8    9   10

Frequently Asked Questions

Q1: Can I nest more than two for loops in R?
Yes, you can nest as many for loops as needed, but readability and performance may suffer with deep nesting .

Q2: Are nested for loops slow in R?
They can be, especially for large datasets. Prefer vectorized functions or parallel processing when possible .

Q3: What’s the difference between break and next?
break exits the innermost loop, while next skips to the next iteration of the innermost loop .

Q4: How do I avoid growing objects inside loops?
Preallocate your output (e.g., use matrix(0, n, m)) before the loop starts.

Q5: When should I use apply instead of nested for loops?
Use apply or similar functions for operations that can be vectorized, especially on matrices or data frames, for better performance .

Key Takeaways

• Nested for loops are essential for multi-dimensional iteration in R.
• Always preallocate output objects for efficiency.
• Use break and next wisely for control flow.
• Prefer vectorized or apply functions when possible.
• Avoid common pitfalls like growing objects inside loops.

Conclusion

Nested for loops are a powerful tool in R programming, enabling you to tackle complex data manipulation and analysis tasks. By mastering their syntax, understanding best practices, and knowing when to use alternatives, you’ll write cleaner, faster, and more reliable R code. Ready to level up your R skills? Start experimenting with nested for loops in your own projects today!

Enjoyed this guide?
Share your feedback below and connect with fellow R programmers on social media. Happy coding!

References

1. An Introduction to R (CRAN Manual)
2. Best Coding Practices for R – Chapter 14: For Loops
3. How to Create a Nested For Loop in R? - GeeksforGeeks
4. Nesting foreach loops (CRAN Vignette)

Happy Coding! 🚀

You can connect with me at any one of the below:

Telegram Channel here: https://t.me/steveondata

LinkedIn Network here: https://www.linkedin.com/in/spsanderson/

Mastadon Social here: https://mstdn.social/@stevensanderson

RStats Network here: https://rstats.me/@spsanderson

GitHub Network here: https://github.com/spsanderson

Bluesky Network here: https://bsky.app/profile/spsanderson.com

My Book: Extending Excel with Python and R here: https://packt.link/oTyZJ

You.com Referral Link: https://you.com/join/EHSLDTL6

@online{
  \_20251103, author = {Sanderson II MPH, Steven P.},
  title = {How to Create a Nested For Loop in R (Including Examples)},
  date = {2025-11-03},
  url = {https://www.spsanderson.com/steveondata/posts/2025-11-03/},
  langid = {en}
  }

Introduction: Why RAG, Ollama, and ragnar in R?

I want to show you how to leverage the power of Retrieval-Augmented Generation (RAG) in R using the Ollama and ragnar packages. This came about by the need for a more efficient way to process and summarize large volumes of text data, particularly in the healthcare domain that I ran into.

Retrieval-Augmented Generation (RAG) is transforming how we interact with documents by combining the strengths of Large Language Models (LLMs) with precise, document-grounded retrieval. With the rise of local LLM runners like Ollama and R packages such as ragnar and ellmer, R programmers can now build powerful, private, and production-ready document summarization pipelines—entirely within R.

This post walks you through a real-world RAG workflow for summarizing health insurance policy PDFs, explaining each code step in simple terms. Whether you’re new to LLMs or looking to automate document Q&A in R, this guide is for you.

Prerequisites & Setup

Before diving in, ensure you have the following:

Tip:
Make sure the Ollama server is running and the required models are pulled before starting your R workflow.

Understanding the RAG Workflow in R

Here’s a high-level overview of the RAG pipeline you’ll build:

1. Load and list PDF files
2. Convert documents to markdown and chunk them
3. Generate embeddings using Ollama
4. Store and index chunks in a DuckDB vector database
5. Register retrieval as a tool for the LLM
6. Query the LLM for summaries, grounded in your documents
7. Format and send results via email or markdown

Step-by-Step Code Walkthrough

Let’s break down the workflow, explaining each step and key syntax.

library(ragnar)
library(ellmer)
library(fs)
library(tidyverse)
library(glue)
library(blastula)
library(RDCOMClient)

policy_files_path <- "W:/Path/To/PDFs/"
policy_files <- list.files(policy_files_path, full.names = TRUE)

TOPIC: RAG with Ollama and ragnar in R
KEYWORD: RAG, Ollama, LLM
AUDIENCE: R Programmers
WORDCOUNT: 750 to 1000

Use the following R code as the basis for the blog post. Explain syntax in simple terms and use tables and bullet points where appropriate.

<code>
# Load Libraries ----
library(ragnar)
library(ellmer)
library(fs)
library(tidyverse)
library(glue)
library(blastula)
library(RDCOMClient)


# Files ----
policy_files_path <- "W:/Path/To/PDFs/"
policy_files <- list.files(policy_files_path, full.names = TRUE)

# System Prompt ----
system_prompt <- str_squish(
  "
  You are an expert assistant that summarizes **Health Insurance Payer Policies** clearly and accurately for healthcare, billing, and administrative users.

  When responding, you should first quote relevant material from the documents in the store,
  provide links to the sources, and then add your own context and interpretation. Try to be as concise
  as you are thorough.

  For every document passed to you the output should if applicable include:

  1. Policy Summary: 1–2 paragraphs describing purpose, scope, and coverage intent.
  2. Key Points: At least 3 concise bullet points summarizing coverage criteria, limitations, exclusions, or authorization requirements.
  3. Policy Information Table

  **Model Behavior Rules:**

  * If information is missing, state “Not specified in document.”
  * Do not infer or assume; summarize only verifiable content.
  * Maintain neutral, factual tone using payer-standard language (e.g., “medically necessary,” “experimental/investigational”).
  * Simplify complex clinical text while preserving accuracy.
  * Always follow the structure: **Policy Summary → Key Points → Policy Information Table.**
  * Avoid opinion, speculation, or advice; ensure compliance-focused clarity.
  "
)

file_split_tbl <- tibble(
  file_path = policy_files
) |>
  mutate(
    file_name = path_file(file_path),
    file_extension = path_ext(file_path),
    file_size = file_size(file_path),
    file_date = file_info(file_path)$modification_time
  ) |>
  group_split(file_name)

llm_resp_list <- file_split_tbl |>
  imap(
    .f = function(obj, id) {
      file_path <- obj |> pull(1) |> pluck(1)
      store_location <- "pdf_ragnar_duckdb"
      store <- ragnar_store_create(
        store_location,
        embed = \(x) embed_ollama(x, model = "nomic-embed-text:latest"),
        overwrite = TRUE
      )
      chunks <- file_path |> read_as_markdown() |> markdown_chunk()
      ragnar_store_insert(store, chunks)
      ragnar_store_build_index(store)
      client <- chat_ollama(
        model = "gpt-oss:20b-cloud",
        system_prompt = system_prompt,
        params = list(temperature = 0.1)
      )
      ragnar_register_tool_retrieve(chat = client, store = store)
      user_prompt <- glue("Please summarize the policy: {file_path}")
      res <- client$chat(user_prompt, echo = "all")
      rec <- obj |> mutate(llm_resp = res)
      return(rec)
    }
  )

output_tbl <- list_rbind(llm_resp_list) |>
  mutate(
    email_body = md(glue("
      Please see summary for below:

      Name: {file_name}
      
      Extension: {file_extension}
      
      Size: {file_size} bytes
      
      Date: {file_date}
      
      Summary Response: 
      
      {llm_resp}
    "))
  )

walk(
  .x = output_tbl$email_body,
  ~ {
    Outlook <- COMCreate("Outlook.Application")
    Email <- Outlook$CreateItem(0)
    Email[["subject"]] <- "Payer Policy Summary"
    Email[["htmlbody"]] <- markdown::markdownToHTML(.x)
    attachment <- str_replace_all(
      output_tbl$file_path[output_tbl$email_body == .x],
      "/",
      "\\\\"
    )
    Email[["to"]] <- "who gets the summarized files?"
    Email[["attachments"]]$Add(attachment)
    Email$Send()
    rm(Outlook)
    rm(Email)
    Sys.sleep(1)
  }
)

markdown_sections <- map_chr(1:nrow(output_tbl), function(i) {
  row_to_md(output_tbl[i, ])
})
markdown_doc <- paste(markdown_sections, collapse = "\n---\n")
write_file(markdown_doc, paste0(getwd(), "/test_policy_output.md"))

Key Functions Reference

Best Practices & Tips

• Keep it Local: All processing and LLM inference happen on your machine—no sensitive data leaves your environment.
• Use Clear Prompts: A detailed system prompt ensures consistent, compliance-ready summaries.
• Chunk Wisely: Proper chunking improves retrieval accuracy and LLM grounding.
• Monitor Memory: Large PDFs and embeddings can be memory-intensive; process in batches if needed.
• Automate Output: Use email or markdown export to integrate with business workflows.

FAQ: RAG with Ollama and ragnar in R

Q1: What is Retrieval-Augmented Generation (RAG)?
A: RAG combines LLMs with document retrieval, ensuring responses are grounded in your actual data.

Q2: Why use Ollama with R?
A: Ollama lets you run LLMs and embedding models locally, keeping data private and reducing latency.

Q3: What file types are supported?
A: ragnar supports PDF, DOCX, HTML, and more—anything convertible to markdown.

Q4: Can I use other LLMs or embedding models?
A: Yes! ragnar and ellmer support multiple providers (OpenAI, Google, etc.), but Ollama is ideal for local workflows.

Q5: How do I troubleshoot model or memory issues?
A: Ensure Ollama is running, models are pulled, and process large files in smaller batches if you hit memory limits.

Quick Takeaways

• RAG in R is now practical and private with ragnar, ellmer, and Ollama.
• Chunking and embedding are key to accurate, document-grounded LLM responses.
• Automated summarization can be integrated into business workflows via email or markdown.
• All code is modular—customize prompts, models, and output as needed.
• Ideal for sensitive data—no cloud required.

Conclusion & Next Steps

Building a RAG pipeline in R with Ollama and ragnar empowers you to automate document summarization, Q&A, and compliance reporting—entirely on your own infrastructure. This approach is especially valuable for healthcare, legal, and enterprise settings where privacy and accuracy are paramount.

Ready to try it?
Install the packages, set up Ollama, and adapt the code to your own document collections. Explore advanced features like hybrid retrieval, custom chunking, or integrating with Shiny dashboards for interactive Q&A.

Further Reading & Resources

• ragnar documentation
• ellmer documentation
• Ollama official site

Share your feedback!
Did you find this guide helpful? Share your experience or questions in the comments, and connect with the document summarizing!**

Happy Coding! 🚀

Understanding Python’s Time Management Tools

Python provides several modules for working with time and dates. Let’s explore the most important ones for beginners:

import time

# Measure how long code takes to run
start_time = time.time()

# Your code here
end_time = time.time()
print(f"Execution time: {end_time - start_time} seconds")

# Pause execution for 3 seconds
time.sleep(3)
print("This prints after 3 seconds!")

Execution time: 6.842613220214844e-05 seconds
This prints after 3 seconds!

from datetime import datetime, timedelta

# Get current date and time
now = datetime.now()
print(now.strftime("%Y-%m-%d %H:%M:%S"))

# Calculate future dates
tomorrow = now + timedelta(days=1)
next_week = now + timedelta(weeks=1)

2025-10-23 09:38:10

Scheduling Tasks with Python

pip install schedule

import schedule
import time

def backup_files():
    print("Running backup...")
    # Your backup logic here

def send_report():
    print("Sending daily report...")
    # Your reporting logic here

# Schedule tasks
schedule.every().day.at("02:00").do(backup_files)
schedule.every().friday.at("17:00").do(send_report)
schedule.every(30).minutes.do(lambda: print("System check"))

# Keep scheduler running
while True:
    schedule.run_pending()
    time.sleep(1)

import time
from datetime import datetime

def simple_scheduler():
    last_run = time.time()
    interval = 60  # Run every 60 seconds
    
    while True:
        current_time = time.time()
        if current_time - last_run >= interval:
            print(f"Task executed at {datetime.now().strftime('%H:%M:%S')}")
            last_run = current_time
        time.sleep(1)

Launching External Programs

import subprocess

# Run a Python script
result = subprocess.run(['python', 'my_script.py'], 
                       capture_output=True, 
                       text=True)

if result.returncode == 0:
    print("Success:", result.stdout)
else:
    print("Error:", result.stderr)

# ✅ Safe way
subprocess.run(['ls', '-l', '/home/user'])

# ❌ Dangerous way (vulnerable to injection)
subprocess.run("ls -l /home/user", shell=True)

Practical Applications: Putting It All Together

import subprocess
import time
from datetime import datetime
import schedule

def create_backup():
    """Create a timestamped backup"""
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    backup_name = f"backup_{timestamp}.tar.gz"
    
    print(f"Starting backup at {datetime.now().strftime('%H:%M:%S')}")
    
    # Measure backup time
    start_time = time.time()
    
    # Create backup (example command)
    try:
        result = subprocess.run([
            'tar', '-czf', backup_name, '/path/to/data'
        ], capture_output=True, text=True, timeout=300)
        
        end_time = time.time()
        duration = end_time - start_time
        
        if result.returncode == 0:
            print(f"Backup completed in {duration:.2f} seconds")
            log_backup_success(backup_name, duration)
        else:
            print(f"Backup failed: {result.stderr}")
            
    except subprocess.TimeoutExpired:
        print("Backup timed out after 5 minutes")

def log_backup_success(filename, duration):
    """Log successful backup to file"""
    with open('backup_log.txt', 'a') as f:
        timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
        f.write(f"[{timestamp}] Backup created: {filename} ({duration:.2f}s)\n")

# Schedule daily backups
schedule.every().day.at("02:00").do(create_backup)

Your Turn!

Challenge: Create a simple system monitor that checks disk space every 5 minutes and logs the results.

Requirements:

• Use subprocess to run a disk space command
• Schedule the check to run every 5 minutes
• Log results with timestamps
• Display results in a readable format

import subprocess
import time
from datetime import datetime
import schedule

def check_disk_space():
    """Monitor disk space and log results"""
    try:
        # Get disk usage (Unix/Linux command)
        result = subprocess.run(['df', '-h'], 
                              capture_output=True, 
                              text=True, 
                              timeout=10)
        
        if result.returncode == 0:
            timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
            
            # Parse the output
            lines = result.stdout.strip().split('\n')
            root_line = [line for line in lines if line.endswith('/')][0]
            usage = root_line.split()[4]  # Get usage percentage
            
            # Log to file
            with open('disk_monitor.log', 'a') as f:
                f.write(f"[{timestamp}] Disk Usage: {usage}\n")
            
            print(f"Disk usage logged: {usage} at {timestamp}")
            
            # Alert if usage is high
            if int(usage.strip('%')) > 80:
                print("⚠️  Warning: Disk usage above 80%!")
                
    except Exception as e:
        print(f"Error checking disk space: {e}")

# Schedule the check
schedule.every(5).minutes.do(check_disk_space)

# Run scheduler
print("Starting disk space monitor...")
while True:
    schedule.run_pending()
    time.sleep(1)

Quick Takeaways

• Use time.time() and time.sleep() for basic timing and delays in your programs
• Master datetime for working with dates, formatting, and date arithmetic
• Install and use the schedule library for easy task automation with human-readable syntax
• Always use subprocess.run() instead of older methods like os.system() for launching programs
• Use command lists, not strings with subprocess for better security
• Capture output with capture_output=True when you need to process program results
• Set timeouts to prevent your programs from hanging on external commands
• Combine these tools to create powerful automation scripts for backups, monitoring, and maintenance

Conclusion

Python’s time management and process control capabilities make it an excellent choice for automation and system administration tasks. By mastering the time and datetime modules for timekeeping, the schedule library for task automation, and the subprocess module for program launching, you’ll be able to build sophisticated scripts that handle complex workflows.

Start small with simple scheduled tasks, then gradually incorporate external program launches as you become more comfortable. The combination of these tools will dramatically expand what you can accomplish with Python automation.

Ready to automate your workflow? Pick one repetitive task you do regularly and try scheduling it with Python. Your future self will thank you for the time saved!

References

1. Python Software Foundation. (2024). time — Time access and conversions. Python 3 Documentation. Retrieved from https://docs.python.org/3/library/time.html

2. Python Software Foundation. (2024). subprocess — Subprocess management. Python 3 Documentation. Retrieved from https://docs.python.org/3/library/subprocess.html

3. Real Python Team. (2024). A Beginner’s Guide to the Python time Module. Real Python. Retrieved from https://realpython.com/python-time-module/

4. schedule Library Contributors. (2024). schedule: Python job scheduling for humans. Schedule Documentation. Retrieved from https://schedule.readthedocs.io/

Happy Coding! 🚀

You can connect with me at any one of the below:

Telegram Channel here: https://t.me/steveondata

LinkedIn Network here: https://www.linkedin.com/in/spsanderson/

Mastadon Social here: https://mstdn.social/@stevensanderson

RStats Network here: https://rstats.me/@spsanderson

GitHub Network here: https://github.com/spsanderson

Bluesky Network here: https://bsky.app/profile/spsanderson.com

My Book: Extending Excel with Python and R here: https://packt.link/oTyZJ

You.com Referral Link: https://you.com/join/EHSLDTL6

The Base R Standard: Using complete.cases()

The most common and often the fastest way to remove rows containing any NA is by leveraging the built-in Base R function, complete.cases(). This method is the workhorse of R data cleaning, offering a balance of speed and simplicity.

# 1. Setup Sample Data
df_test <- data.frame(
    x = c(10, 20, NA, 40, NA),
    y = c(NA, 50, NA, 80, 90),
    z = c(6, 7, NA, 9, 10)
)
print("Original Data Frame:")

[1] "Original Data Frame:"

print(df_test)

   x  y  z
1 10 NA  6
2 20 50  7
3 NA NA NA
4 40 80  9
5 NA 90 10

# 2. Method 1: Remove rows with NA in at least one column
# The comma at the end applies the filter to the rows (before the comma is the row index)
df_cleaned_cc <- df_test[complete.cases(df_test), ]
print("Cleaned using complete.cases():")

[1] "Cleaned using complete.cases():"

print(df_cleaned_cc)

   x  y z
2 20 50 7
4 40 80 9

Addressing Truly Empty Rows with rowSums()

While complete.cases() is great for removing incomplete records, sometimes you only want to remove rows that are entirely empty (i.e., every cell in that row is NA). This requires a slightly different approach using rowSums() and is.na().

This method works by:

1. is.na(df): Converts the data frame into a matrix of logical values (TRUE where there’s an NA).
2. rowSums(...): Sums the TRUE values (which are treated as 1s) across each row, counting the total number of NAs per row.
3. != ncol(df): Filters the rows where the sum of NAs is not equal to the total number of columns. If the row sum of NAs equals the number of columns, the row is 100% empty and is removed.

Example Code:

# Setup data frame where row 3 is fully NA
df_truly_empty <- data.frame(
    x = c(10, NA, NA, 40),
    y = c(50, 60, NA, 80),
    z = c(1, 2, NA, 4)
)
print("Original Data Frame:")

[1] "Original Data Frame:"

print(df_truly_empty)

   x  y  z
1 10 50  1
2 NA 60  2
3 NA NA NA
4 40 80  4

# Method 2: Remove rows where all columns are NA
df_cleaned_rs <- df_truly_empty[rowSums(is.na(df_truly_empty)) != ncol(df_truly_empty), ]
print("Cleaned using rowSums():")

[1] "Cleaned using rowSums():"

print(df_cleaned_rs)

   x  y z
1 10 50 1
2 NA 60 2
4 40 80 4

Notice how row 3 is removed, but row 2 (which had only one NA) is retained.

The Tidyverse Approach: dplyr::drop_na()

For those who prefer the readable, pipe-friendly syntax of the Tidyverse, the tidyr package offers the concise drop_na() function. This method achieves the same result as complete.cases()—removing all rows with at least one missing value.

The primary advantage here is readability; the intent is immediately clear, especially when chaining multiple data operations.

Example Code:

# Ensure dplyr is loaded
library(tidyr)

# Method 3: Remove rows with NA using dplyr
df_cleaned_tidyr <- df_test |>
    drop_na()
print("Cleaned using tidyr::drop_na():")

[1] "Cleaned using tidyr::drop_na():"

print(df_cleaned_tidyr)

   x  y z
1 20 50 7
2 40 80 9

Performance Matters: Benchmarking Removal Methods

As R programmers, we care about more than just syntax—we care about efficiency. While tidyr is often favored for readability, Base R methods can sometimes offer a performance edge on very large datasets.

We use the rbenchmark package to compare the two common methods for removing incomplete rows: Base R complete.cases() and the tidyr::drop_na() function.

library(rbenchmark)

# Create a large test data frame (10,000 rows)
set.seed(42)
big_df <- as.data.frame(
    matrix(
        sample(c(1:100, NA), 10000 * 5, replace = TRUE, prob = c(rep(0.95/100, 100), 0.05)),
        ncol = 5
    )
)

# Benchmark the two common methods
benchmark_results <- benchmark(
  BaseR_Complete = big_df[complete.cases(big_df), ],
  tidyr_DropNa = big_df |> drop_na(),
  replications = 1000,
  columns = c("test", "replications", "elapsed", "relative")
)

print(benchmark_results[order(benchmark_results$relative), ])

            test replications elapsed relative
2   tidyr_DropNa         1000    0.44    1.000
1 BaseR_Complete         1000    1.66    3.773

The benchmarking results consistently show that the Base R complete.cases() method is slower than tidyr::drop_na() when executing the same operation on large data frames. For most day-to-day tasks, this difference is negligible, even though there is a large speedup but for massive datasets or functions running millions of times, tidyr retains a slight performance advantage.

Your Turn!

Imagine you have a data frame of customer survey responses. You want to remove only those rows where both the Satisfaction and Usage columns are missing, allowing rows with NA in only one of them to remain.

Your Task: Write the Base R code to remove rows where Satisfaction and Usage are both NA from the data frame survey_df.

survey_df <- data.frame(
    Customer = 1:5,
    Satisfaction = c(5, NA, 3, NA, 4),
    Usage = c(10, 5, NA, NA, 8)
)

# Logical vector where both are NA
is_double_na <- is.na(survey_df$Satisfaction) & is.na(survey_df$Usage)

# Filter the data frame to keep rows where 'is_double_na' is FALSE
# The '!' negates the logical vector, keeping rows that are NOT double NA
cleaned_survey_df <- survey_df[!is_double_na, ]

print(cleaned_survey_df)

  Customer Satisfaction Usage
1        1            5    10
2        2           NA     5
3        3            3    NA
5        5            4     8

Key Takeaways

• Most Readable: df %>% drop_na() (Tidyverse) offers the clearest, most readable syntax for Tidyverse users.
• Targeted Cleaning: You can use complete.cases(df[ , c("col1", "col2")]) to check for completeness in only a subset of columns.
• Truly Empty Rows: Use the rowSums(is.na(df)) != ncol(df) method to target and remove only rows that are 100% missing.

Frequently Asked Questions (FAQs)

1. What is the difference between complete.cases() and na.omit()?

   • complete.cases() returns a logical vector (TRUE/FALSE) that you can use for flexible subsetting. na.omit() is a high-level function that directly returns the data frame with all incomplete rows removed. For simple cleaning, the results are functionally the same.

2. Can I use drop_na() on specific columns?

   • Yes. You can pass specific column names to the function: df %>% drop_na(column_a, column_b). This removes rows only if they have an NA in the specified columns.

3. Why do my empty rows sometimes show up as "" (empty strings) instead of NA?

   • This is common when reading messy CSVs. If a cell is truly blank, R might import it as "". You must first convert these empty strings to the official NA missing value before using the functions discussed: df[df == ""] <- NA.

4. Is one method always better than the others?

   • No. For raw performance on massive datasets, Base R often wins. For clear, pipe-friendly code in a Tidyverse project, dplyr::drop_na() is preferred. Choose the method that best fits your coding environment and performance needs.

5. Does complete.cases() consider rows with NaN?

   • No, complete.cases() only checks for NA (Not Available). If you need to include NaN (Not a Number, often used in mathematical operations) as a missing value, you should first convert it to NA or use a custom function.

Conclusion: Choosing Your Cleaning Tool

Removing empty or incomplete rows is a foundational skill in R. Whether you are a performance purist favoring the speed of complete.cases() or a Tidyverse enthusiast prioritizing the clarity of dplyr::drop_na(), R provides the perfect tool for your data cleaning toolkit.

Choosing between them ultimately boils down to balancing performance benchmarks with code readability. Start cleaning your data frames today for more accurate, robust statistical models!

What’s your go-to method? Share your preferred data cleaning function in the comments below!

References

• Statology. (n.d.). How to Remove Empty Rows from Data Frame in R. Retrieved from https://www.statology.org/remove-empty-rows-in-r/
• Wickham, H., François, R., Henry, L., & Müller, K. (2024). dplyr: A Grammar of Data Manipulation. R package version 1.1.4.
• R Core Team (2024). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/.

Happy Coding! 🚀

Sample Data Setup

Let’s start with a sample dataset to demonstrate each method:

# Sample data
data <- data.frame(
  ID = c(1, 2, 3, 4, 5),
  Value = c(10, 25, 15, 25, 20),
  Group = c("A", "A", "B", "B", "C")
)

print(data)

  ID Value Group
1  1    10     A
2  2    25     A
3  3    15     B
4  4    25     B
5  5    20     C

Base R Methods

# Returns first occurrence only
data[which.max(data$Value), ]

  ID Value Group
2  2    25     A

# Returns all rows with max value
data[data$Value == max(data$Value), ]

  ID Value Group
2  2    25     A
4  4    25     B

dplyr Methods

The dplyr package offers slice_max() for intuitive max row selection .

library(dplyr)

# First max row only
data %>% slice_max(Value, n = 1, with_ties = FALSE)

  ID Value Group
1  2    25     A

# All rows with max value (default behavior)
data %>% slice_max(Value, n = 1)

  ID Value Group
1  2    25     A
2  4    25     B

# Max row per group
data %>% group_by(Group) %>% slice_max(Value, n = 1)

# A tibble: 3 × 3
# Groups:   Group [3]
     ID Value Group
  <dbl> <dbl> <chr>
1     2    25 A    
2     4    25 B    
3     5    20 C    

data.table Methods

data.table provides the fastest performance for large datasets .

library(data.table)
dt <- as.data.table(data)

# First max row
dt[which.max(Value)]

      ID Value  Group
   <num> <num> <char>
1:     2    25      A

# All max rows
dt[Value == max(Value)]

      ID Value  Group
   <num> <num> <char>
1:     2    25      A
2:     4    25      B

# Max row per group (fastest method)
dt[, .SD[which.max(Value)], by = Group]

    Group    ID Value
   <char> <num> <num>
1:      A     2    25
2:      B     4    25
3:      C     5    20

Performance Comparison Using rbenchmark

Row selection (filtering) is one of the most common data manipulation operations. Let’s compare the performance of Base R, dplyr, and data.table for filtering operations using the rbenchmark package .

# Install and load required packages
library(rbenchmark)
library(data.table)
library(dplyr)

# Create sample data
set.seed(123)
n <- 100000L
dt <- data.table(
  id = 1:n,
  category = sample(c("A", "B", "C", "D"), n, replace = TRUE),
  value = rnorm(n, mean = 50, sd = 15),
  flag = sample(c(TRUE, FALSE), n, replace = TRUE)
)

# Convert to data.frame for base R and dplyr
df <- as.data.frame(dt)

# Benchmark: Finding rows with maximum value
result <- benchmark(
  base_R = df[df$value == max(df$value), ],
  data_table = dt[value == max(value)],
  dplyr = filter(df, value == max(value)),
  replications = 1000,
  columns = c("test", "replications", "elapsed", "relative")
) |>
        arrange(relative)

print(result)

        test replications elapsed relative
1     base_R         1000    1.72    1.000
2      dplyr         1000    4.25    2.471
3 data_table         1000    6.10    3.547

Handling Edge Cases

# Base R: Remove NAs
data[data$Value == max(data$Value, na.rm = TRUE), ]

  ID Value Group
2  2    25     A
4  4    25     B

# dplyr: Filter NAs first
data %>% filter(!is.na(Value)) %>% slice_max(Value, n = 1)

  ID Value Group
1  2    25     A
2  4    25     B

Method Selection Guide

Your Turn!

Now it’s your turn to experiment with rbenchmark and deepen your understanding of R performance optimization!

library(stringr)

# Create test data
text_data <- data.frame(
  id = 1:50000,
  text = sample(c("apple", "banana", "cherry", "date", "elderberry"), 
                50000, replace = TRUE),
  stringsAsFactors = FALSE
)

# Your task: Benchmark these approaches for finding rows containing "app"
benchmark(
  base_R = text_data[grepl("app", text_data$text), ],
  base_R_fixed = text_data[grepl("app", text_data$text, fixed = TRUE), ],
  stringr = text_data[str_detect(text_data$text, "app"), ],
  replications = 50,
  columns = c("test", "elapsed", "relative")
) |>
        arrange(relative)

          test elapsed relative
1 base_R_fixed    0.26    1.000
2       base_R    0.95    3.654
3      stringr    1.30    5.000

# Create grouping data
group_data <- data.frame(
  group = sample(LETTERS[1:10], 10000, replace = TRUE),
  value = rnorm(10000)
)
dt_group <- as.data.table(group_data)

# Your task: Benchmark mean calculation by group
benchmark(
  base_R = aggregate(value ~ group, data = group_data, FUN = mean),
  data_table = dt_group[, .(mean_value = mean(value)), by = group],
  dplyr = group_data %>% group_by(group) %>% summarise(mean_value = mean(value)),
  replications = 100,
  columns = c("test", "elapsed", "relative")
) |>
        arrange(relative)

        test elapsed relative
1 data_table    0.23    1.000
2      dplyr    0.67    2.913
3     base_R    1.23    5.348

# Your task: Compare memory efficiency
library(pryr)  # for object_size()

# Create copies for fair comparison
df_copy <- df
dt_copy <- copy(dt)

# Time and measure memory for column addition
system.time({
  df_result <- transform(df_copy, new_col = value * 2)
  cat("Base R result size:", object_size(df_result), "\n")
})

Base R result size: 3201456 

   user  system elapsed 
   0.02    0.00    0.03 

system.time({
  dt_copy[, new_col := value * 2]
  cat("data.table result size:", object_size(dt_copy), "\n")
})

data.table result size: 3602088 

   user  system elapsed 
      0       0       0 

Quick Takeaways

Some quick easy takeaways from this guide:

• Base R which.max() and logical subsetting provide simple, dependency-free solutions • dplyr slice_max() offers the most readable syntax with excellent tie handling • data.table delivers superior performance, especially for large datasets and grouped operations • rbenchmark helps you make data-driven decisions about method selection • Always consider handling NA values and ties in real-world applications • Choose your method based on dataset size, performance requirements, and code readability preferences • Performance differences become more significant with larger datasets and complex operations

Conclusion

Selecting rows with maximum values in R is straightforward with all three approaches. Base R methods work well for most scenarios without additional packages. dplyr excels when code readability matters most. data.table is your best choice for performance-critical applications with large datasets.

The rbenchmark package provides valuable insights into actual performance differences, helping you make informed decisions about which method to use for your specific situation .

Your turn: Try implementing these methods with your own data and compare the performance differences. Start with the approach that best fits your current workflow and data size requirements, then optimize based on your benchmarking results!

References

1. Sanderson, S. (2024, December 10). How to Select Row with Max Value in Specific Column in R: A Complete Guide. R-bloggers. https://www.r-bloggers.com/2024/12/how-to-select-row-with-max-value-in-specific-column-in-r-a-complete-guide/

2. Statology. (n.d.). R: How to Select Row with Max Value in Specific Column. Statology. Retrieved October 6, 2025, from https://www.statology.org/r-select-row-with-max-value/

Happy Coding! 🚀

You can connect with me at any one of the below:

Telegram Channel here: https://t.me/steveondata

LinkedIn Network here: https://www.linkedin.com/in/spsanderson/

Mastadon Social here: https://mstdn.social/@stevensanderson

RStats Network here: https://rstats.me/@spsanderson

GitHub Network here: https://github.com/spsanderson

Bluesky Network here: https://bsky.app/profile/spsanderson.com

My Book: Extending Excel with Python and R here: https://packt.link/oTyZJ

You.com Referral Link: https://you.com/join/EHSLDTL6

Introduction

Python offers straightforward ways to handle CSV and JSON files, two common formats for storing and exchanging data. This guide shows how to read, write, and convert between these formats using Python’s built-in tools and popular libraries.

Reading and Writing CSV Files

Python’s csv module provides simple methods for CSV operations:

import csv

# Reading a CSV file
with open('data.csv') as file:
    reader = csv.reader(file)
    for row in reader:
        print(row)

# Writing to CSV
with open('output.csv', 'w') as file:
    writer = csv.writer(file)
    writer.writerow(['Name', 'Age'])
    writer.writerow(['Alice', 30])

For more advanced data handling, the pandas library simplifies CSV operations:

import pandas as pd

# Read CSV into a DataFrame
data = pd.read_csv('data.csv')

# Write DataFrame to CSV
data.to_csv('output.csv', index=False)

Working with JSON Data

Python’s json module makes JSON operations simple:

import json

# Reading JSON
with open('data.json') as file:
    data = json.load(file)

# Writing JSON
with open('output.json', 'w') as file:
    json.dump(data, file, indent=4)

Converting Between CSV and JSON

Converting between formats is straightforward:

CSV to JSON:

import csv, json

csv_data = []
with open('data.csv') as file:
    reader = csv.DictReader(file)
    for row in reader:
        csv_data.append(row)

with open('output.json', 'w') as file:
    json.dump(csv_data, file, indent=4)

JSON to CSV:

import csv, json

with open('data.json') as file:
    json_data = json.load(file)

with open('output.csv', 'w') as file:
    writer = csv.DictWriter(file, fieldnames=json_data[0].keys())
    writer.writeheader()
    writer.writerows(json_data)

Your Turn!

1. Create a CSV file with sample data (name, email, age)
2. Write a Python script to convert it to JSON
3. Modify the JSON data by adding a new field
4. Convert the modified JSON back to CSV

# Step 1: Create sample CSV
import csv
data = [['name','email','age'],
        ['Alice','alice@example.com',28],
        ['Bob','bob@example.com',35]]
with open(r'C:\Users\steve\Documents\GitHub\steveondata\posts\2025-10-01\sample.csv', 'w') as f:
    writer = csv.writer(f)
    writer.writerows(data)

# Step 2: Convert to JSON
import json
with open(r'C:\Users\steve\Documents\GitHub\steveondata\posts\2025-10-01\sample.csv') as f:
    reader = csv.DictReader(f)
    rows = list(reader)
with open(r'C:\Users\steve\Documents\GitHub\steveondata\posts\2025-10-01\sample.json', 'w') as f:
    json.dump(rows, f, indent=2)

# Step 3: Add new field
with open(r'C:\Users\steve\Documents\GitHub\steveondata\posts\2025-10-01\sample.json') as f:
    data = json.load(f)
for item in data:
    item['status'] = 'active'
with open(r'C:\Users\steve\Documents\GitHub\steveondata\posts\2025-10-01\updated.json', 'w') as f:
    json.dump(data, f, indent=2)

# Step 4: Convert back to CSV
with open(r'C:\Users\steve\Documents\GitHub\steveondata\posts\2025-10-01\updated.json') as f:
    data = json.load(f)
with open(r'C:\Users\steve\Documents\GitHub\steveondata\posts\2025-10-01\final.csv', 'w') as f:
    writer = csv.DictWriter(f, fieldnames=data[0].keys())
    writer.writeheader()
    writer.writerows(data)

Key Points

• The csv module handles basic CSV operations
• pandas provides advanced CSV capabilities
• JSON works naturally with Python dictionaries
• Conversion between formats is simple with the right tools

References

• Python csv module documentation - Official documentation for Python’s built-in CSV handling capabilities
• Python json module documentation - Official guide for working with JSON data in Python
• pandas read_csv documentation - Complete reference for reading CSV files into DataFrames
• pandas to_csv documentation - Detailed guide for writing DataFrames to CSV files

Happy Coding! 🚀

You can connect with me at any one of the below:

Telegram Channel here: https://t.me/steveondata

LinkedIn Network here: https://www.linkedin.com/in/spsanderson/

Mastadon Social here: https://mstdn.social/@stevensanderson

RStats Network here: https://rstats.me/@spsanderson

GitHub Network here: https://github.com/spsanderson

Bluesky Network here: https://bsky.app/profile/spsanderson.com

My Book: Extending Excel with Python and R here: https://packt.link/oTyZJ

You.com Referral Link: https://you.com/join/EHSLDTL6

Introduction

Data preprocessing is a necessary step in any machine learning workflow. The healthyR.ai package offers user-friendly functions for imputation (filling missing values) and scaling (normalizing data) that integrate seamlessly with the tidymodels ecosystem in R. This guide explains the syntax and implementation of these functions in straightforward terms.

Data Imputation with hai_data_impute()

Imputation replaces missing values in your dataset. The hai_data_impute() function supports multiple imputation methods through a consistent interface.

hai_data_impute(
  .recipe_object = NULL,
  ...,
  .type_of_imputation = "mean",
  .seed_value = 123,
  # Additional parameters based on method
)

library(healthyR.ai)
library(recipes)
library(dplyr)

# Create recipe object
rec_obj <- recipe(value ~ ., df_tbl)

# Apply rolling median imputation
imputed_data <- hai_data_impute(
  .recipe_object = rec_obj,
  value,
  .type_of_imputation = "roll",
  .roll_statistic = median
)$impute_rec_obj %>%
  get_juiced_data()

Data Scaling with hai_data_scale()

Scaling transforms variables to a common scale, which is important for many machine learning algorithms.

hai_data_scale(
  .recipe_object = NULL,
  ...,
  .type_of_scale = "center",
  .range_min = 0,
  .range_max = 1,
  .scale_factor = 1
)

library(healthyR.ai)
library(recipes)
library(dplyr)

# Create recipe object
rec_obj <- recipe(value ~ ., df_tbl)

# Apply standardization (z-score)
scaled_data <- hai_data_scale(
  .recipe_object = rec_obj,
  value,
  .type_of_scale = "scale"
)$scale_rec_obj %>%
  get_juiced_data()

Combining Imputation and Scaling

A typical preprocessing workflow combines both steps:

# Create recipe
rec_obj <- recipe(target ~ ., data_df)

# Step 1: Impute missing values
imputed <- hai_data_impute(
  .recipe_object = rec_obj,
  all_numeric(),
  .type_of_imputation = "median"
)$impute_rec_obj

# Step 2: Scale the imputed data
final_data <- hai_data_scale(
  .recipe_object = imputed,
  all_numeric(),
  .type_of_scale = "range"
)$scale_rec_obj %>%
  get_juiced_data()

Exampls

I think things work best when you can see an example in action.

library(healthyR.ai)
library(recipes)
library(dplyr)
library(ggplot2)
library(purrr)

n <- 10L
l <- 5L
lo <- n * l

date_seq <- seq.Date(from = as.Date("2013-01-01"), length.out = lo, by = "month")
date_seq

 [1] "2013-01-01" "2013-02-01" "2013-03-01" "2013-04-01" "2013-05-01"
 [6] "2013-06-01" "2013-07-01" "2013-08-01" "2013-09-01" "2013-10-01"
[11] "2013-11-01" "2013-12-01" "2014-01-01" "2014-02-01" "2014-03-01"
[16] "2014-04-01" "2014-05-01" "2014-06-01" "2014-07-01" "2014-08-01"
[21] "2014-09-01" "2014-10-01" "2014-11-01" "2014-12-01" "2015-01-01"
[26] "2015-02-01" "2015-03-01" "2015-04-01" "2015-05-01" "2015-06-01"
[31] "2015-07-01" "2015-08-01" "2015-09-01" "2015-10-01" "2015-11-01"
[36] "2015-12-01" "2016-01-01" "2016-02-01" "2016-03-01" "2016-04-01"
[41] "2016-05-01" "2016-06-01" "2016-07-01" "2016-08-01" "2016-09-01"
[46] "2016-10-01" "2016-11-01" "2016-12-01" "2017-01-01" "2017-02-01"

val_seq <- replicate(n = l, c(runif(9), NA)) |> as.vector() |> as.double()
val_seq

 [1] 0.74815520 0.62345014 0.98719405 0.98357823 0.64343460 0.38288945
 [7] 0.30782868 0.63132596 0.09734484         NA 0.79572696 0.08743225
[13] 0.72841099 0.78703884 0.39553790 0.54639674 0.96807028 0.60125354
[19] 0.74665373         NA 0.92237646 0.04457192 0.68444841 0.05388607
[25] 0.24374963 0.73552094 0.84926348 0.55056715 0.77699405         NA
[31] 0.55460139 0.24564446 0.24396533 0.60797386 0.71226179 0.93048958
[37] 0.72179306 0.01549613 0.88487496         NA 0.41888816 0.08623630
[43] 0.06213051 0.58266383 0.72425739 0.17659346 0.80285097 0.78684451
[49] 0.68433082         NA

data_tbl <- tibble(
  date_col = date_seq,
  value = val_seq
)

rec_obj <- recipe(value ~ date_col, data = data_tbl)
rec_obj

df_tbl <- tibble(
  impute_type = c("bagged","knn","linear","mean","median","roll"),
  rec_obj = list(rec_obj),
  data = list(data_tbl)
)
df_tbl[1,][[3]][[1]]

# A tibble: 50 × 2
   date_col     value
   <date>       <dbl>
 1 2013-01-01  0.748 
 2 2013-02-01  0.623 
 3 2013-03-01  0.987 
 4 2013-04-01  0.984 
 5 2013-05-01  0.643 
 6 2013-06-01  0.383 
 7 2013-07-01  0.308 
 8 2013-08-01  0.631 
 9 2013-09-01  0.0973
10 2013-10-01 NA     
# ℹ 40 more rows

data_list <- df_tbl |>
  group_split(impute_type)

data_impute_list <- data_list |>
  imap(
    .f = function(obj, id){
      imp_type = obj |> pull(impute_type)
      rec_obj = obj |> pull(rec_obj) |> pluck(1)
      data = obj[["data"]][[1]]
      
      imp_obj <- hai_data_impute(
        .recipe_object = rec_obj,
        value,
        .type_of_imputation = imp_type,
        .roll_statistic = median
      )$impute_rec_obj

      imputed_data <- get_juiced_data(imp_obj)

      combined_tbl <- data |>
        left_join(imputed_data, by = "date_col") |>
        setNames(c("date_col", "original_value", "imputed_value")) |>
        mutate(rec_no = row_number()) |>
        mutate(color_col = original_value,
              size_col = original_value) |>
        mutate(impute_type = imp_type)
      
      return(combined_tbl)
    }
  )

combined_tbl <- data_impute_list |>
  list_rbind()

imped_na_vals_tbl <- combined_tbl |>
  filter(is.na(original_value)) |>
  summarize(
        avg_imputed_val = mean(imputed_value),
        .by = impute_type
  )

combined_tbl |>
  summarize(
        avg_imputed_val_col = mean(imputed_value),
        avg_original_val_col = mean(original_value, na.rm = TRUE),
        .by = impute_type
  ) |>
mutate(imputation_diff = avg_imputed_val_col - avg_original_val_col) |>
left_join(imped_na_vals_tbl, by = "impute_type")

# A tibble: 6 × 5
  impute_type avg_imputed_val_col avg_original_val_col imputation_diff
  <chr>                     <dbl>                <dbl>           <dbl>
1 bagged                    0.556                0.559        -0.00287
2 knn                       0.557                0.559        -0.00199
3 linear                    0.558                0.559        -0.00128
4 mean                      0.559                0.559         0      
5 median                    0.566                0.559         0.00721
6 roll                      0.555                0.559        -0.00434
# ℹ 1 more variable: avg_imputed_val <dbl>

ggplot(data = combined_tbl,
  aes(
    x = date_col,
    y = imputed_value,
    color = color_col
    )
  ) + 
  facet_wrap(~ impute_type) +
  geom_point(data = combined_tbl |> filter(is.na(original_value)), aes(shape = 'NA', size = 3)) +
  scale_shape_manual(values = c('NA' = 3)) +
  geom_line(aes(x = date_col, y = original_value), color = "black") +
  geom_line(aes(x = date_col, y = imputed_value), color = "red", linetype = "dashed", alpha = .328) +
  geom_vline(
    data = combined_tbl[combined_tbl$original_value |> is.na(), ], 
    aes(xintercept = date_col), color = "black", linetype = "dashed"
  ) +
  labs(
    x = "Date",
    y = "Value",
    title = "Original vs. Imputed Data using HealthyR.ai",
    subtitle = "Function: hai_data_impute()",
    caption = "Red line is the imputed data, blue line is the original data"
  ) +
  theme_classic() +
  theme(legend.position = "none")

combined_tbl |>
  filter(is.na(original_value)) |>
  ggplot(aes(x = impute_type, y = imputed_value, color = impute_type, group = impute_type)) +
  geom_boxplot() +
  labs(
    x = "Date",
    y = "Value",
    title = "Original vs. Imputed Data using HealthyR.ai",
    subtitle = "Function: hai_data_impute()"
  ) +
  theme_classic() +
  theme(legend.position = "none")

Choosing the Right Method

For imputation:

• Continuous normal data: Use “mean”
• Skewed data or outliers: Use “median”
• Categorical data: Use “mode”
• Time series: Use “roll” with appropriate window
• Complex relationships: Try “knn” or “bagged”

For scaling:

• Linear regression: Use “center” or “scale”
• Distance-based algorithms (KNN, SVM): Use “scale”
• Neural networks: Use “range” [0,1] or “normalize”

Best Practices

• Always load required libraries: healthyR.ai, recipes, dplyr
• Create recipe object first: recipe(target ~ ., data = df)
• Set .seed_value for reproducible results with stochastic methods
• Extract processed data with get_juiced_data() function
• Verify results with summary() after processing
• Choose imputation method based on data characteristics and missingness pattern

Key Takeaways

• healthyR.ai provides user-friendly wrappers around recipes functions for data preprocessing
• Both imputation and scaling require a recipe object
• Functions return a list containing the processed recipe object
• Choose methods based on your data type and the requirements of your modeling approach
• Chain operations (imputation → scaling → modeling) for a complete workflow

The goal of these healthyR.ai functions is to simplify data preprocessing with a consistent syntax and integration with the tidymodels ecosystem, making it an excellent choice for streamlining your machine learning pipeline.

Happy Coding! 🚀

You can connect with me at any one of the below:

Telegram Channel here: https://t.me/steveondata

LinkedIn Network here: https://www.linkedin.com/in/spsanderson/

Mastadon Social here: https://mstdn.social/@stevensanderson

RStats Network here: https://rstats.me/@spsanderson

GitHub Network here: https://github.com/spsanderson

Bluesky Network here: https://bsky.app/profile/spsanderson.com

My Book: Extending Excel with Python and R here: https://packt.link/oTyZJ

You.com Referral Link: https://you.com/join/EHSLDTL6

What Can You Accomplish?

With Python’s document manipulation capabilities, you can:

• 📄 Extract text from PDF files (like meeting minutes or reports)
  
• 🔗 Combine multiple PDF files into one document
  
• 🔄 Rotate pages that were scanned sideways
  
• 📝 Create new Word documents automatically
  
• ✏️ Read and modify existing Word files
  
• 🔒 Handle password-protected PDFs

Getting Started: Installing the Libraries

Before diving into code, you’ll need to install two essential libraries. Open your command prompt or terminal and type:

pip install PyPDF2==1.26.0
pip install python-docx

Working with PDF Files Using PyPDF2

import PyPDF2
import os

# Get current working directory
print("Current Working Directory:", os.getcwd())

# Open the PDF file in binary read mode
pdfFileObj = open('some_pdf.pdf', 'rb')
pdfReader = PyPDF2.PdfFileReader(pdfFileObj)

# Get the first page (Python counts from 0)
pageObj = pdfReader.getPage(0)

# Extract the text
text = pageObj.extractText()
print(text)

# Always close the file
pdfFileObj.close()

import PyPDF2

pdfFileObj = open('some_pdf.pdf', 'rb')
pdfReader = PyPDF2.PdfFileReader(pdfFileObj)
print(f"Number of pages: {pdfReader.numPages}")
pdfFileObj.close()

import PyPDF2

# Open both PDF files
pdf1File = open('january_report.pdf', 'rb')
pdf2File = open('february_report.pdf', 'rb')

# Create readers for each file
pdf1Reader = PyPDF2.PdfFileReader(pdf1File)
pdf2Reader = PyPDF2.PdfFileReader(pdf2File)

# Create a writer to build the new PDF
pdfWriter = PyPDF2.PdfFileWriter()

# Add all pages from first PDF
for pageNum in range(pdf1Reader.numPages):
    pageObj = pdf1Reader.getPage(pageNum)
    pdfWriter.addPage(pageObj)

# Add all pages from second PDF
for pageNum in range(pdf2Reader.numPages):
    pageObj = pdf2Reader.getPage(pageNum)
    pdfWriter.addPage(pageObj)

# Save the combined PDF
pdfOutputFile = open('combined_reports.pdf', 'wb')
pdfWriter.write(pdfOutputFile)
pdfOutputFile.close()
pdf1File.close()
pdf2File.close()

Working with Word Documents Using python-docx

import docx

# Create a new document
document = docx.Document()

# Add a heading
document.add_heading('My Automated Report', 0)

# Add a paragraph
document.add_paragraph('This document was created automatically with Python!')

# Save the document
document.save(r'C:/Users/ssanders/Documents/GitHub/steveondata/posts/2025-09-24/my_report.docx')

import docx

# Open an existing document
doc = docx.Document(r'C:/Users/ssanders/Documents/GitHub/steveondata/posts/2025-09-24/my_report.docx')

# Print each paragraph
for para in doc.paragraphs:
    print(para.text)

My Automated Report
This document was created automatically with Python!

from docx import Document

# Open the template
doc = Document(r'C:/Users/ssanders/Documents/GitHub/steveondata/posts/2025-09-24/my_report.docx')

# Add new content
doc.add_paragraph('New agenda item added automatically.')

# Save with a new name
doc.save(r'C:/Users/ssanders/Documents/GitHub/steveondata/posts/2025-09-24/updated_my_report.docx')

Practical Real-World Applications

Your Turn! Practice Exercise

Challenge: Create a Python script that:

1. Opens a Word document
2. Adds today’s date as a heading
3. Adds a paragraph with your name
4. Saves it as a new file

from docx import Document
from datetime import date

# Create new document
doc = Document()

# Add today's date as heading
today = date.today()
doc.add_heading(f'Report for {today}', 0)

# Add paragraph with name
doc.add_paragraph('Created by: [Your Name Here]')

# Save the document
doc.save(f'C:/Users/ssanders/Documents/GitHub/steveondata/posts/2025-09-24/daily_report_{today}.docx')

print("Document created successfully!")

Document created successfully!

Key Takeaways

Quick Reference Points:

• Use PyPDF2 for reading and manipulating PDF files
• Use python-docx for creating and editing Word documents
• Always open PDFs in binary mode ('rb')
• Remember Python uses zero-based indexing (first page is 0)
• Close files after opening them to free up memory
• These tools are perfect for automating repetitive tasks

What’s Next?

Try these examples with your own PDF and Word files. Start small - maybe extract text from a single PDF or create a simple Word document. Once you’re comfortable, you can combine these techniques to build powerful document automation tools.

The beauty of working with PDF and Word documents in Python is that once you master these basics, you can automate almost any document-related task!

References

• Automate the Boring Stuff with Python, Chapter 15: Working with PDF and Word Documents
• python-docx Official Documentation
• PyPDF2 Documentation
• Automate the Boring Stuff with Python - Main Site
• Installing Third-Party Modules Guide

Happy Coding! 🚀

You can connect with me at any one of the below:

Telegram Channel here: https://t.me/steveondata

LinkedIn Network here: https://www.linkedin.com/in/spsanderson/

Mastadon Social here: https://mstdn.social/@stevensanderson

RStats Network here: https://rstats.me/@spsanderson

GitHub Network here: https://github.com/spsanderson

Bluesky Network here: https://bsky.app/profile/spsanderson.com

My Book: Extending Excel with Python and R here: https://packt.link/oTyZJ

You.com Referral Link: https://you.com/join/EHSLDTL6

Understanding Row-Wise Operations in R

Row-wise operations in R work across the columns of each row, rather than down the columns. When we want the max value in each row, we’re looking for the highest value across all columns for every single row in our dataset.

Basic Concept:

• Column-wise: Operations down each column (like finding the mean of each column)
• Row-wise: Operations across columns for each row (like finding the max of each row)

Method 1: Using apply() - The Most Common Approach

The apply() function is the most popular method for finding row maximums in R. It’s part of base R, so no additional packages are required.

apply(X, MARGIN, FUN, ...)

# Create sample data
df <- data.frame(
  A = c(1, 4, 7, 2, 9),
  B = c(2, 5, 8, 6, 3),
  C = c(3, 6, 9, 4, 1),
  D = c(5, 2, 1, 8, 7)
)

# Find max in each row
df$max_value <- apply(df, 1, max)
print(df)

  A B C D max_value
1 1 2 3 5         5
2 4 5 6 2         6
3 7 8 9 1         9
4 2 6 4 8         8
5 9 3 1 7         9

# Data with missing values
df_na <- data.frame(
  A = c(1, 4, NA, 2, 9),
  B = c(2, NA, 8, 6, 3),
  C = c(3, 6, 9, NA, 1),
  D = c(5, 2, 1, 8, 7)
)

# Use na.rm = TRUE to ignore missing values
df_na$max_value <- apply(df_na, 1, max, na.rm = TRUE)
print(df_na)

   A  B  C D max_value
1  1  2  3 5         5
2  4 NA  6 2         6
3 NA  8  9 1         9
4  2  6 NA 8         8
5  9  3  1 7         9

Method 2: Using pmax() - Fastest for Few Columns

The pmax() function computes parallel maximums, making it excellent for datasets with a small number of columns .

# Using pmax for the same data
df$max_pmax <- pmax(df$A, df$B, df$C, df$D)

# Alternative with do.call
df$max_pmax2 <- do.call(pmax, df[1:4])
print(df)

  A B C D max_value max_pmax max_pmax2
1 1 2 3 5         5        5         5
2 4 5 6 2         6        6         6
3 7 8 9 1         9        9         9
4 2 6 4 8         8        8         8
5 9 3 1 7         9        9         9

Method 3: Using dplyr - Tidyverse Approach

For those using the tidyverse, dplyr offers modern, readable approaches to row-wise operations .

library(dplyr)

# Method 3a: Using rowwise() and c_across()
df <- df %>%
  rowwise() %>%
  mutate(max_tidy = max(c_across(A:D))) %>%
  ungroup()

# Method 3b: Using pmax within mutate
df <- df %>%
  mutate(max_pmax_tidy = pmax(A, B, C, D))

print(df)

# A tibble: 5 × 9
      A     B     C     D max_value max_pmax max_pmax2 max_tidy max_pmax_tidy
  <dbl> <dbl> <dbl> <dbl>     <dbl>    <dbl>     <dbl>    <dbl>         <dbl>
1     1     2     3     5         5        5         5        5             5
2     4     5     6     2         6        6         6        6             6
3     7     8     9     1         9        9         9        9             9
4     2     6     4     8         8        8         8        8             8
5     9     3     1     7         9        9         9        9             9

Method 4: Using matrixStats - Best for Large Datasets

For large datasets, the matrixStats package provides optimized functions that significantly outperform base R alternatives .

library(matrixStats)

# Convert to matrix and use rowMaxs
df$max_matrixstats <- rowMaxs(as.matrix(df[1:4]))
glimpse(df)

Rows: 5
Columns: 10
$ A               <dbl> 1, 4, 7, 2, 9
$ B               <dbl> 2, 5, 8, 6, 3
$ C               <dbl> 3, 6, 9, 4, 1
$ D               <dbl> 5, 2, 1, 8, 7
$ max_value       <dbl> 5, 6, 9, 8, 9
$ max_pmax        <dbl> 5, 6, 9, 8, 9
$ max_pmax2       <dbl> 5, 6, 9, 8, 9
$ max_tidy        <dbl> 5, 6, 9, 8, 9
$ max_pmax_tidy   <dbl> 5, 6, 9, 8, 9
$ max_matrixstats <dbl> 5, 6, 9, 8, 9

Method Comparison Table

Practical Examples

# Student performance data
exam_scores <- data.frame(
  Student = c("Alice", "Bob", "Charlie", "Diana", "Eve"),
  Math = c(85, 92, 78, 95, 88),
  Science = c(90, 85, 82, 93, 91),
  English = c(88, 89, 85, 97, 86),
  History = c(82, 88, 90, 89, 94)
)

# Find highest score for each student
exam_scores$Highest_Score <- apply(exam_scores[2:5], 1, max)
print(exam_scores)

  Student Math Science English History Highest_Score
1   Alice   85      90      88      82            90
2     Bob   92      85      89      88            92
3 Charlie   78      82      85      90            90
4   Diana   95      93      97      89            97
5     Eve   88      91      86      94            94

# Quarterly stock prices with missing data
stock_prices <- data.frame(
  Stock = c("AAPL", "GOOGL", "MSFT", "AMZN", "TSLA"),
  Q1 = c(150.5, 2800.0, NA, 3200.0, 800.0),
  Q2 = c(NA, 2750.0, 280.0, 3100.0, 750.0),
  Q3 = c(145.0, NA, 285.0, 3250.0, 900.0),
  Q4 = c(160.0, 2900.0, 290.0, NA, 850.0)
)

# Find maximum price per stock
stock_prices$Max_Price <- apply(stock_prices[2:5], 1, max, na.rm = TRUE)

# Find which quarter had the max price
stock_prices$Best_Quarter <- apply(stock_prices[2:5], 1, function(x) {
  names(x)[which.max(x)]
})

glimpse(stock_prices)

Rows: 5
Columns: 7
$ Stock        <chr> "AAPL", "GOOGL", "MSFT", "AMZN", "TSLA"
$ Q1           <dbl> 150.5, 2800.0, NA, 3200.0, 800.0
$ Q2           <dbl> NA, 2750, 280, 3100, 750
$ Q3           <dbl> 145, NA, 285, 3250, 900
$ Q4           <dbl> 160, 2900, 290, NA, 850
$ Max_Price    <dbl> 160, 2900, 290, 3250, 900
$ Best_Quarter <chr> "Q4", "Q4", "Q4", "Q3", "Q3"

Common Pitfalls and Solutions

• Forgetting na.rm = TRUE: Returns NA if any value in row is missing - Solution: Always use na.rm = TRUE when dealing with missing data

• Wrong MARGIN parameter: Using MARGIN = 2 finds column max, not row max - Solution: Remember 1 = rows, 2 = columns

• All-NA rows: With na.rm = TRUE, returns -Inf instead of NA - Solution: Use custom function to check for all-NA rows

• Character columns: max() doesn’t work on text data - Solution: Select only numeric columns first

Your Turn!

Practice Exercise:

Create a data frame with sales data for different products across four months and find the best performing month for each product.

# Your challenge data
sales_data <- data.frame(
  Product = c("Laptop", "Phone", "Tablet", "Watch"),
  Jan = c(1200, 800, 600, 300),
  Feb = c(1100, 850, 550, 350),
  Mar = c(1300, 900, 700, 400),
  Apr = c(1250, 820, 650, 380)
)

# TODO: Find the maximum sales for each product
# TODO: Find which month had the highest sales for each product

# Solution 1: Find maximum sales
sales_data$Best_Sales <- apply(sales_data[2:5], 1, max)

# Solution 2: Find best month
sales_data$Best_Month <- apply(sales_data[2:5], 1, function(x) {
  names(x)[which.max(x)]
})

# Alternative using multiple methods
sales_data$Max_pmax <- pmax(sales_data$Jan, sales_data$Feb, 
                           sales_data$Mar, sales_data$Apr)

print(sales_data)

  Product  Jan  Feb  Mar  Apr Best_Sales Best_Month Max_pmax
1  Laptop 1200 1100 1300 1250       1300        Mar     1300
2   Phone  800  850  900  820        900        Mar      900
3  Tablet  600  550  700  650        700        Mar      700
4   Watch  300  350  400  380        400        Mar      400

Key Takeaways

• apply(df, 1, max) is the most common and versatile method for finding row maximums • Use na.rm = TRUE when your data contains missing values • pmax() is faster for datasets with few columns • matrixStats::rowMaxs() provides the best performance for large datasets • dplyr::rowwise() integrates well with tidyverse workflows • Always specify MARGIN = 1 in apply() for row-wise operations

Conclusion

Finding the max value in each row in R can be accomplished through multiple approaches, each with specific advantages. The apply() function remains the gold standard for most users due to its simplicity and reliability. For performance-critical applications, consider matrixStats::rowMaxs(), while tidyverse users will appreciate dplyr’s readable syntax.

Choose the method that best fits your workflow, data size, and coding style. With these techniques, you’ll efficiently handle row-wise maximum calculations in any R project.

Ready to level up your R skills? Try implementing these methods with your own datasets and see which approach works best for your specific use case!

References

1. How to Find the Max Value in Each Row in R
2. Row-wise operations (dplyr documentation)
3. Apply Functions Over Array Margins (R Documentation)

Happy Coding! 🚀

You can connect with me at any one of the below:

Telegram Channel here: https://t.me/steveondata

LinkedIn Network here: https://www.linkedin.com/in/spsanderson/

Mastadon Social here: https://mstdn.social/@stevensanderson

RStats Network here: https://rstats.me/@spsanderson

GitHub Network here: https://github.com/spsanderson

Bluesky Network here: https://bsky.app/profile/spsanderson.com

My Book: Extending Excel with Python and R here: https://packt.link/oTyZJ

You.com Referral Link: https://you.com/join/EHSLDTL6

What is EZSheets?

Google Sheets is a free, web-based spreadsheet application that’s perfect for collaboration. EZSheets is a Python library that makes working with Google Sheets incredibly simple . Think of it as a translator between Python and Google Sheets - it handles all the complex API details so you can focus on your data.

Unlike the official Google Sheets API (which can be overwhelming for beginners), EZSheets uses straightforward syntax that feels natural to Python programmers.

Installation and Setup

pip install ezsheets

Your First Google Sheets Script

Let’s start with a simple example from “Automate the Boring Stuff”:

import ezsheets

# This will open a browser window for login (first time only)
ss = ezsheets.createSpreadsheet('My First Python Spreadsheet')
print(f"Created spreadsheet: {ss.title}")
print(f"Available sheets: {ss.sheetTitles}")

What’s happening here:

• import ezsheets: Loads the EZSheets library
• createSpreadsheet(): Makes a new Google Sheet with the name you choose
• ss.title: Gets the name of your spreadsheet
• ss.sheetTitles: Shows all the worksheet tabs (like “Sheet1”)

Understanding Spreadsheet Structure

Think of Google Sheets like a filing cabinet :

• Spreadsheet: The entire file (like a binder)
• Sheet/Worksheet: Individual tabs within the file (like pages in the binder)
• Cells: Individual boxes where data goes (like A1, B2, C3)

Basic Operations

# Method 1: By spreadsheet ID (from the URL)
ss = ezsheets.Spreadsheet('your-spreadsheet-id-here')

# Method 2: Upload a local file
ss = ezsheets.upload('my_excel_file.xlsx')

# Get the first sheet
sheet = ss[0]  # Using index number

# Get sheet by name  
sheet = ss['Sheet1']  # Using sheet name

# See all available sheets
print(ss.sheetTitles)  # Returns: ('Sheet1', 'Sheet2', ...)

# Get all data from a sheet
all_data = sheet.getRows()  # Returns a list of lists
print(all_data)

# Access specific cells
first_cell = sheet[1, 1]  # Row 1, Column 1 (A1)
print(f"Cell A1 contains: {first_cell}")

# Using A1 notation (like in Excel)
cell_value = sheet['A1']
print(f"A1 value: {cell_value}")

# Update a single cell
sheet['A1'] = 'Hello, World!'

# Update an entire row
sheet.updateRow(1, ['Name', 'Age', 'City', 'Occupation'])
sheet.updateRow(2, ['Alice', 25, 'New York', 'Engineer'])

# Update multiple cells
sheet['B1'] = 'Age'
sheet['C1'] = 'City'

Common Use Cases

# Example: Adding survey results
import ezsheets

ss = ezsheets.createSpreadsheet('Survey Results')
sheet = ss[0]

# Set up headers
headers = ['Participant', 'Rating', 'Comments', 'Date']
sheet.updateRow(1, headers)

# Add data
survey_data = [
    ['John Doe', 5, 'Excellent service!', '2024-01-15'],
    ['Jane Smith', 4, 'Very good experience', '2024-01-16']
]

for i, row_data in enumerate(survey_data, start=2):  # Start from row 2
    sheet.updateRow(i, row_data)

# Read existing data for analysis
ss = ezsheets.Spreadsheet('your-existing-spreadsheet-id')
sheet = ss['Sales Data']

all_rows = sheet.getRows()
header = all_rows[0]  # First row is usually headers
data_rows = all_rows[1:]  # Everything except the header

print(f"Found {len(data_rows)} records")
print(f"Columns: {', '.join(header)}")

Your Turn!

Now it’s time to put your Google Sheets automation skills into practice! Choose one of these beginner-friendly exercises to get started:

import ezsheets

# Step 1: Create or connect to your spreadsheet
ss = ezsheets.createSpreadsheet('Sales Report Practice')
sheet = ss[0]

# Step 2: Set up headers
headers = ['Date', 'Product', 'Quantity', 'Price', 'Total']
sheet.updateRow(1, headers)

# Step 3: Add sample data
sample_data = [
    ['2024-01-15', 'Widget A', 10, 25.99, ''],
    ['2024-01-16', 'Widget B', 5, 45.50, ''],
    ['2024-01-17', 'Widget C', 8, 19.99, ''],
    ['2024-01-18', 'Widget A', 15, 25.99, ''],
    ['2024-01-19', 'Widget D', 3, 75.00, '']
]

# Add each row of data
for i, row in enumerate(sample_data, 2):  # Start at row 2
    sheet.updateRow(i, row)

print(f"Exercise complete! View your sheet at: {ss.url}")

import ezsheets

def create_sales_report():
    # Create new spreadsheet
    ss = ezsheets.createSpreadsheet('Sales Report Practice')
    sheet = ss[0]
    
    # Set up headers
    headers = ['Date', 'Product', 'Quantity', 'Price', 'Total']
    sheet.updateRow(1, headers)
    
    # Sample sales data
    sales_data = [
        ['2024-01-15', 'Widget A', 10, 25.99],
        ['2024-01-16', 'Widget B', 5, 45.50],
        ['2024-01-17', 'Widget C', 8, 19.99],
        ['2024-01-18', 'Widget A', 15, 25.99],
        ['2024-01-19', 'Widget D', 3, 75.00]
    ]
    
    # Add data and calculate totals
    total_sales = 0
    for i, (date, product, qty, price) in enumerate(sales_data, 2):
        total = qty * price
        total_sales += total
        sheet.updateRow(i, [date, product, qty, price, total])
    
    # Add summary row
    sheet.updateRow(7, ['TOTAL', '', '', '', total_sales])
    
    # Format currency columns (Price and Total)
    # Note: EZSheets has limited formatting - for advanced formatting, use gspread
    
    print(f"Sales report created successfully!")
    print(f"Total Sales: ${total_sales:.2f}")
    print(f"View your sheet: {ss.url}")
    
    return ss.url

# Run the function
if __name__ == "__main__":
    create_sales_report()

import ezsheets

def create_inventory_tracker():
    # Create new spreadsheet
    ss = ezsheets.createSpreadsheet('Inventory Tracker')
    sheet = ss[0]
    
    # Set up headers
    headers = ['Item', 'Current Stock', 'Minimum Threshold', 'Status']
    sheet.updateRow(1, headers)
    
    # Sample inventory data
    inventory = [
        ['Laptop', 15, 10],
        ['Mouse', 8, 15],     # This will be low stock
        ['Keyboard', 25, 20],
        ['Monitor', 5, 12],   # This will be low stock
        ['Webcam', 30, 10],
        ['Headset', 18, 15]
    ]
    
    low_stock_items = []
    
    # Process each item
    for i, (item, current, minimum) in enumerate(inventory, 2):
        # Determine status
        if current < minimum:
            status = "Low Stock"
            low_stock_items.append(item)
        else:
            status = "In Stock"
        
        # Update the row
        sheet.updateRow(i, [item, current, minimum, status])
    
    # Add summary section
    sheet.updateRow(len(inventory) + 3, ['SUMMARY:', '', '', ''])
    sheet.updateRow(len(inventory) + 4, ['Low Stock Items:', len(low_stock_items), '', ''])
    
    # List low stock items
    for j, item in enumerate(low_stock_items):
        sheet.updateRow(len(inventory) + 5 + j, [item, 'Needs Restocking', '', ''])
    
    print(f"Inventory tracker created!")
    print(f"Items needing restock: {', '.join(low_stock_items) if low_stock_items else 'None'}")
    print(f"View your sheet: {ss.url}")
    
    return ss.url

# Run the function
if __name__ == "__main__":
    create_inventory_tracker()

Go To Commands Reference