Pandas and Python Tips and Tricks for Data Science and Data Analysis
Take your efficiency to the next level with these Pandas and Python Tricks!
Motivation
This blog regroups all the Pandas and Python tricks & tips I share on a basis on my LinkedIn page. I have decided to centralize them into a single blog to help you make the most out of your learning process by easily finding what you are looking for.
The content is is divided into two main sections:
- Pandas tricks & tips are related to only Pandas.
- Python tricks & tips related to Python.
Pandas tricks & tips
This section provides a list of all the tricks
1. ðð¿ð²ð®ðð² ð® ð»ð²ð ð°ðŒð¹ððºð» ð³ð¿ðŒðº ðºðð¹ðð¶ðœð¹ð² ð°ðŒð¹ððºð»ð ð¶ð» ððŒðð¿ ð±ð®ðð®ð³ð¿ð®ðºð².
Performing simple arithmetic tasks such as creating a new column as the sum of two other columns can be straightforward.
ð€ But, what if you want to implement a more complex function and use it as the logic behind column creation? Here is where things can get a bit challenging.
Guess whatâŠ
â ðð¥ð¥ð¡ð® and ð¡ðð¢ððð can help you easily apply whatever logic to your columns using the following format:
ðð[ð£ðð¬_ðð€ð¡] = ðð.ðð¥ð¥ð¡ð®(ð¡ðð¢ððð ð§ð€ð¬: ððªð£ð(ð§ð€ð¬), ððððš=1)
where:
â¡ ðð is your dataframe.
â¡ ð§ð€ð¬ will correspond to each row in your data frame.
â¡ ððªð£ð is the function you want to apply to your data frame.
â¡ ððððš=1 to apply the function to each row in your data frame.
ð¡ Below is an illustration.
The `candidate_info` function combines each candidateâs information to create a single description column about that candidate.
2. Convert categorical data into numerical ones
This process mainly can occur in the feature engineering phase. Some of its benefits are:
- the identification of outliers, invalid, and missing values in the data.
- reduction of the chance of overfitting by creating more robust models.
â¡ Use these two functions from Pandas, depending on your need. Examples are provided in the image below.
1ïžâ£ .ððªð©() to specifically define your bin edges.
ðððð£ðð§ðð€
Categorize candidates by expertise with respect to their number of experience, where:
- Entry level: 0â1 year
- Mid-level: 2â3 years
- Senior level: 4â5 years
2ïžâ£ .ðŠððªð©() to divide your data into equal-sized bins.
It uses the underlying percentiles of the distribution of the data, rather than the edges of the bins.
ðððð£ðð§ðð€: categorize the commute time of the candidates into ðð€ð€ð, ððððð¥ð©ððð¡ð, or ð©ð€ð€ ð¡ð€ð£ð.
ðððð¥ ðð£ ð¢ðð£ð ð¡
- When using .ððªð©(): a number of bins = number of labels + 1.
- When using .ðŠððªð©(): a number of bins = number of labels.
- With .ððªð©(): set ðð£ðð¡ðªðð_ð¡ð€ð¬ððšð©=ðð§ðªð, otherwise, the lowest value will be converted to NaN.
3. Select rows from a Pandas Dataframe based on column(s) values
â¡ use .ðŠðªðð§ð®() function by specifying the filter condition.
â¡ the filter expression can contain any operators (<, >, ==, !=, etc.)
â¡ use the @Ì· sign to use a variable in the expression.
4. Deal with zip files
Sometimes it can be efficient to read and write .zip files without extracting them from your local disk. Below is an illustration.
5. Select ð® ððð¯ðð²ð ðŒð³ ððŒðð¿ ð£ð®ð»ð±ð®ð ð±ð®ðð®ð³ð¿ð®ðºð² ðð¶ððµ ððœð²ð°ð¶ð³ð¶ð° ð°ðŒð¹ððºð» ðððœð²ð
You can use the ðšðð¡ððð©_ðð©ð®ð¥ððš function. It takes two main parameters: ððððððð ððð ðð¡ððððð.
- ðð.ðððððð_ððð¢ððð(ððððððð = [âðð¢ðð_ð·â, âðð¢ðð_ðžâ, ⊠âðð¢ðð_ðâ]) means I want the subset of my data frame WITH columns of ðð¢ðð_ð·, ðð¢ðð_ðž,âŠ, ðð¢ðð_ð.
- ðð.ðððððð_ððð¢ððð(ðð¡ððððð = [âðð¢ðð_ð·â, âðð¢ðð_ðžâ, ⊠âðð¢ðð_ðâ]) means I want the subset of my data frame WITHOUT columns of ðð¢ðð_ð·, ðð¢ðð_ðž,âŠ, ðð¢ðð_ð.
âš Below is an illustration
6. Remove comments from Pandas dataframe column
Imagine that I want clean this data (candidates.csv) by removing comments from the application date column. This can be done on the fly while loading your pandas dataframe using the ðð€ð¢ð¢ðð£ð© parameter as follow:
â¡ ððððð_ðððð = ðð.ðððð_ððð(ðððð_ðð_ðððð, ðð€ð¢ð¢ðð£ð©=âðð¢ððððâ)
In my case, ðð€ð¢ð¢ðð£ð©=â#â but it could be any other character (|, /, etc.) depending on your case. An illustration is the first scenario.
âðœ Wait, what if I want to create a new column for those comments and still remove them from the application date column? An illustration is the second scenario.
7. Print Pandas dataframe in Tabular format from consol
â No, the application of the ððððð() function to a pandas data frame does not always render an output that is easy to read, especially for data frames with multiple columns.
â
If you want to get a nice console-friendly tabular output
Use the .ðð_ðððððð() function as illustrated below.
8. Highlight data points in Pandas
Applying colors to a pandas data frame can be a good way to emphasize certain data points for quick analysis.
â This is where ðððððð.ððð¢ðð module comes in handy. It has many features, but is not limited to the followings:
âš ðð.ððð¢ðð.ððððððððð_ððð¡() to assign a color to the maximum value of each column.
âš ðð.ððð¢ðð.ððððððððð_ðin() to assign a color to the minimum value of each column.
âš ðð.ððð¢ðð.ððððð¢(ðð¢_ðððððð_ðððððððð) to apply your custom function to your data frame.
9. Reduce decimal points in your data
Sometimes, very long decimal values in your data set do not provide significant information and can be painful ð€¯ to look at.
So, you might want to convert your data to about 2 to 3 decimal points to facilitate your analysis.
â This is something you can perform using the ðððððð.ð³ððððµðððð.ððððð() function as illustrated below.
10. Replace some values in your data frame
You might want to replace some information in your data frame to keep it as up-to-date as possible.
â This can be achieved using the Pandas ððððððððð.ððððððð() function as illustrated below.
11. Compare two data frames and get their differences
Sometimes, when comparing two pandas data frames, not only do you want to know if they are equivalent, but also where the difference lies if they are not equivalent.
â This is where the .ððððððð() function comes in handy.
âš It generates a data frame showing columns with differences side by side. Its shape is different from (0, 0) only if the two data being compared are the same.
âš If you want to show values that are equal, set the ðððð_ððððð parameter to ðððð. Otherwise, they are shown as ðœððœ.
12. Get a subset of a very large dataset for quick analysis
Sometimes, we just need a subset of a very large dataset for quick analysis. One of the approaches could be to read the whole data in memory before getting your sample.
This can require a lot of memory depending on how big your data is. Also, it can take significant time to read your data.
â You can use ðððð ð parameter in the pandas ðððð_ððð() function by specifying the number of rows you want.
13. Transform your data frame from a wide to a long format
Sometimes it can be useful ððððððððð ð¢ððð ððððððððð ðððð ð ð ððð ðð ð ðððð ðððððð which is more flexible for better analysis, especially when dealing with time series data.
- ðððð© ðð€ ð®ð€ðª ð¢ððð£ ðð® ð¬ððð & ð¡ð€ð£ð?
âš Wide format is when you have a lot of columns.
âš Long format on the other side is when you have a lot of rows.
â ð¿ððððð.ðððð() is a perfect candidate for this task.
Below is an illustration
14. Reduce the size of your Pandas data frame by ignoring the index
Do you know that you can reduce the size of your Pandas data frame by ignoring the index when saving it?
â Something like ððððð¡ = ðµðððð when saving the file.
Below is an illustration.
15. Parquet instead of CSV
Very often, I donât manually look ð at the content of a CSV or Excel file that will be used by Pandas for further analysis.
If thatâs your case, maybe you should not use .CSV anymore and think of a better option.
Especially if you are only concerned about
âš Processing speed
âš Speed in saving and loading
âš Disk space occupied by the data frame
â In that case, .ð¥ðð§ðŠðªðð© format is your best option as illustrated below.
16. Transform your data frame into a markdown
It is always better to print your data frame in a way that makes it easier to understand.
â One way of doing that is to render it in a markdown format using the .ðð_ððððððð ð() function.
ð¡ Below is an illustration
17. Format Date Time column
When loading Pandas dataframes, date columns are represented as ðŒð¯ð·ð²ð°ð by default, which is not â the correct date format.
â You can specify the target column in the ðœð®ð¿ðð²_ð±ð®ðð²ð argument to get the correct column type.
Python tips and tricks
1. Create a progress bar with tqdm and rich
Using the progress bar is beneficial when you want to have a visual status of a given task.
#!pip -q install rich
from rich.progress import track
from tqdm import tqdm
import time
Implement the callback function
def compute_double(x):
return 2*x
Create the progress bars
2. Get day, month, year, day of the week, the month of the year
3. Smallest and largest values of a column
If you want to get the rows with the largest or lowest values for a given column, you can use the following functions:
âš ðð.ðððððððð(ðœ, âð²ðð_ðœðððâ) â top ðœ rows based on ð²ðð_ðœððð
âš ðð.ððððððððð(ðœ, âð²ðð_ðœðððâ) â ðœ smallest rows based on ð²ðð_ðœððð
âš ð²ðð_ðœððð is the name of the column you are interested in.
4. Ignore the log output of the pip install command
Sometimes when installing a library from your jupyter notebook, you might not want to have all the details about the installation process generated by the default ððð ððððððð command.
â You can specify the -q or â quiet option to get rid of that information.
Below is an illustration ð¡
5. Run multiple commands in a single notebook cell
The exclamation mark â!â is essential to successfully run a shell command from your Jupyter notebook.
However, this approach can be quite repetitive ð when dealing with multiple commands or a very long and complicated one.
â A better way to tackle this issue is to use the %%ððð¬ð¡ expression at the beginning of your notebook cell.
ð¡ Below is an illustration
6. Virtual environment.
A Data Science project can involve multiple dependencies, and dealing with all of them can be a bit annoying. ð€¯
âš A good practice is to organize your project in a way that it can be easily shared with your team members and reproduced with the least amount of effort.
â One way of doing this is to use virtual environments.
âïž ðð¿ð²ð®ðð² ðð¶ð¿ððð®ð¹ ð²ð»ðð¶ð¿ðŒð»ðºð²ð»ð ð®ð»ð± ð¶ð»ððð®ð¹ð¹ ð¹ð¶ð¯ð¿ð®ð¿ð¶ð²ð.
â Install the virtual environment module.
ððð ððððððð ðððððððððð
â Create your environment by giving a meaningful name.
ðððððððððð [ð¢ððð_ððððððððððð_ðððð]
â Activate your environment.
ðððððð [ð¢ððð_ððððððððððð_ðððð]/ððð/ðððððððð
â Start installing the dependencies for your project.
ððð ððððððð ðððððð
âŠ
All this is great ððŒ, BUT⊠the virtual environment you just created is local to your machineð.
ðððð© ð©ð€ ðð€?ð€·ð»ââïž
ð¡ You need to permanently save those dependencies in order to share them with others using this command:
â ððð ððððð£ð > ðððððððððððð.ðð¡ð
This will create ðððððððððððð.ðð¡ð file containing your project dependencies.
ð Finally, anyone can install the exact same dependencies by running this command:
â ððð ððððððð -ð ðððððððððððð.ðð¡ð
7. Run multiple metrics at once
Scikit learn metrics
8. Chain multiple lists as a single sequence
You can use a single for loop to iterate through multiple lists as a single sequence ð.
â This can be achieved using the ððððð() â function from Python ð¶ðð²ð¿ððŒðŒð¹ð module.
9. Pretty print of JSON data
â Have ever wanted to print your JSON data in a correct indented format for better visualization?
â The indent parameter of the dumps() method can be used to specify the indentation level of your formatted string output.
Conclusion
Thank you for reading! ð ðŸ
I hope you found this list of Python and Pandas tricks helpful! Keep an eye on here, because the content will be maintained with more tricks on a daily basis.
Also, If you like reading my stories and wish to support my writing, consider becoming a Medium member. With a $ 5-a-month commitment, you unlock unlimited access to stories on Medium.
Feel free to follow me on Medium, Twitter, and YouTube, or say Hi on LinkedIn. It is always a pleasure to discuss AI, ML, Data Science, NLP, and MLOps stuff!

















































