Other Pylint warnings related to the logging module are:

• logging-not-lazy / W1201

• logging-format-interpolation / W1202

Pylint

A little introduction about pylint. Pylint is a static code analyzer, it analyses your code without actually running it. Pylint looks for potential errors, gives suggestions on coding standards that your code is not adhering to, potential places where refactoring might help, and also warnings about smelly code.

Sometimes there are false positives, but you can inspect and decide for yourself whether it flagged the line incorrectly or not. If you identify it as a false positive, you can disable the pylint check for that particular line.

Overall, I think it takes out a lot of pain from the reviewer to point out code smells and enforce coding standards. I would say this module has saved hours of review time.

logging-fstring-interpolation warning

Python has multiple ways of string interpolation. I have explained the details of it in a previous blog - String Interpolation in Python

Even though we are using the latest string interpolation mechanism in python, the Pylint checker is giving us a warning that there is something wrong. The official documentation for this warning says:

Use another type of string formatting instead. You can use % formatting but leave interpolation to the logging function by passing the parameters as arguments.

To understand what was happening behind the hood, I wrote this code.

Here, I have an Employee class with an __str__ magic method. The logging module is configured to log into the console and the logging level is set to DEBUG. The statements of interest are all debug statements, and I also added some headings to understand what section we are executing now.

Running this code with the logging level set to DEBUG. I expect to see the str method called for all the cases because the log level is DEBUG is the lowest and all the log messages will be printed.

The response is as expected. The string evaluation of the object takes place and then the log statement is evaluated.

What if the log level is set to INFO, but the log statements are at the DEBUG level?
I expect the string evaluation to be deferred until the logger determines whether this log statement is above the log level set, i.e., all the statements with log level as INFO and higher will be printed whereas all the statements with log level as DEBUG will be ignored. If the log statements with DEBUG are ignored, then there is no need to call the str method of the objects.

Making only 1 change on line #5, I executed the code.

The response was not something I was expecting.

Upon inspection, it was clear the evaluation of the message objects was happening irrespective of whether the statement was executed or discarded.

So irrespective of what type of string interpolation I use, the order of events is as follows:

1. evaluate all the arguments to the log.debug() - this calls the str method,

2. Call the log.debug(),

3. Determine whether to log it or discard it.

This seemed very counterintuitive because then the logging statement when turned off will still consume time and memory. I decided to peek into the logging library. Looking at the function signature it was clear, there is nothing wrong with the library. But I am not using the library in the intended way.

The intended way to use the logger function is to pass the string template as the first argument and the arguments to be substituted as *args. This way, the string interpolation happens within the logger library and not when I call the logger method.

The correct way to invoke the logging statement where the string interpolation is deferred until it is necessary.

Added two more lines of code and repeated the experiment. In this case, I am passing the %-formatted string template separately and the argument to substitute it with separately.

Running this code again, the result is as expected, the string evaluation is not happening because the log level is set to INFO.

After changing the log level to debug on line #5, the log statement is getting printed as expected.

So, the order of events will be:

1. Call the log.debug(),

2. Determine whether to log it or discard it.

3. Discards the log because the log level is below the level set.

4. Evaluate the message object by substituting the arguments.

Now that the internal workings and the correct usage is clear, let us understand the impact of this small change on our program.

Impact

You might think this string interpolation is harmless, but if the logging statement is in a function that is called frequently and the logging level is set to be discarded during normal operation, you are wasting away a lot of execution cycles. To understand the impact, I looped the logging statements 100,000 times, collected the run time, and tabulated it.

The x-axis defines different approaches and the y-axis gives the time in milliseconds for each approach.

When we are logging and the log message evaluation is expensive, using the approach of deferred evaluation can save a lot of time. When you want to turn off the debug logs and run in optimized mode. Otherwise, all the approaches are yielding similar run time.

Conclusion

Hope this blog helps someone who is scratching their head trying to figure out why the Pylint-type checker is giving this weird warning. Pylint is trying to tell you to check your syntax because you could be committing a mistake by not using deferred evaluation. This warning message is generated when you are trying to use f-string substitution in the logging statement.

Instead of simply disabling the warning, now you know the reason behind it and how to handle it the correct way.

Share with me your thoughts on this in the comments! Or leave a like if you learned something new.

Python has three ways of string interpolation. I am going to explain each of these with one or two examples.

f-strings

One of the latest and most developer-friendly approaches is to use the f-strings. F-strings are a concise and efficient way to embed expressions inside string literals.

name = 'John'
age = 25
print(f'My name is {name} and I am {age} years old.')

Let's look at a more complicated example. Having a list comprehension inside of the string interpolation.

vowels = ('a', 'e', 'i', 'o', 'u')
print(f"These are vowels in English: {', '.join(vowels)}")

One thing to note in the above example is, if you have single quotes in the f-string, the outer quotes cannot be single quotes.

str.format()

The format method of the string class is another way to perform string interpolation. It works by putting one or more placeholders in the string and the values to substitute these placeholders with should be passed as arguments to the format method.

name = 'John'
age = 25
print('My name is {} and I am {} years old.'.format(name, age))

One alternative syntax to use the str.format() is to have named placeholders like in the example below and pass keyworded arguments to replace the placeholders. In the below example, I have to replace the name in two places, but I can get away by passing it only once to the string.

name = 'John'
age = 25
print('My name is {name} and I am {age} years old. \
    My grandfather was also named {name}'.format(name=name, age=age))

I can also store the string with the placeholders as a template somewhere and perform string substitution on demand.

%-formatting

The last mechanism of formatting a Python string is to use the % operator. This syntax feels similar to the printf style in C. Here, the character after the % tells the compiler what the intended value is to be substituted in its place. Check out the example below.

name = 'John'
age = 25
print('My name is %s and I am %d years old.' % (name, age))

Use the below table as a guide for some of the common % symbols. The full list of symbols is available here.

Conclusion

There are also Template strings in Python. I did not cover it in this list of interpolation techniques because it is a vast topic of its own.

Out of the three techniques I like to use f-strings the most followed by %-formatting style when I am using logging statements. The str.format() style is preferred when I have to store the string to interpolate separately and inject the values to the placeholders as keyworded arguments. I use these three techniques during coding depending on the scenario.

I would love to hear if I have missed any other techniques or if you guys have any preferences.

During my research, I stumbled upon the job description "Product Manager". I am collating all my learnings here hoping it might help someone.

What is a product manager?

In a broad sense, you can consider the product manager as the CEO of the product. This term was coined by Ben Horowitz, CEO of Opsware. A product manager is responsible for setting up the vision of how the product should be and what the product will not be. A product manager prepares strategies that strike a balance between the budget, time to market, and resources available. They also have to motivate the execution team and are responsible for the outcome of the development lifecycle.

Does that mean product managers have special authority? They don't.

Responsibilities of a Product Manager

Specific responsibilities vary between organizations, but if I had to summarize my research:

• Identify the customer needs and the larger business objectives a product will fulfill through market research and competitor analysis,

• Define the success metrics for the particular product, and,

• Rally a team to turn that vision into reality.

In a smaller organization, you may have to do more work from defining the vision and seeing it through the execution. But in larger organizations, they may have more sub-teams and tools to make your life easier but aligning everyone to a single vision might take a lot of effort.

Skillsets

The product manager lies at the intersection of business goals, the tech, and the user experience as you can see from the Venn diagram.

Even though there are dedicated teams for UX, Tech, and business, he should know the terms and the lay of the land to communicate effectively with these teams. For example, knowing Figma can make brainstorming with the UX designers more productive.

If you already have the skillsets or are willing to cultivate these, then you can be a good product manager:

1. Good at prioritizing,

2. Learn to influence people without authority,

3. Decision-making on the go,

4. Know the team you're working with.

All these and a knowledge of a lot of tools to help organize your work - JIRA, Confluence, Trello, Teams, Figma, etc.

Salary Scales for Product Managers

All the glam and glory that the product manager role brings apart, what does it pay at the end of the day? Based on my research, the average salary for a product manager in India is about 20 LPA and going up to 35 LPA (at the time of writing this blog). Check here. Largely varying based on your experience and the type of organization.

Conclusion

Doesn't matter whether you are from a service industry, a product company, or starting your own business, a product manager or knowledge of product management is a game changer for the organization. That one person who can envision and rally a team behind that vision will determine whether your product is the go-to product or will be just another product in the market.

After the excitement of using a new system died out, we identified there are issues. I want to talk about one such issue we faced and discuss a solution.

The Problem Statement

We were facing two main issues during the development phase.

1. Managing the database schema:

   We were unable to test out feature branches independently when the feature being implemented contained database schema changes. When such changes came up, we had to serialize the implementation so that other teams are unaffected by this change.

2. Migrating the database into production:

   When we rolled out the changes into production, it was a cumbersome process. Creating a lot of custom scripts to migrate the user database which was part of version A to the target version. These became largely invalid and not maintained, adding to the technical debt.

The solution that I am going to discuss will solve the below problems:

1. Bring up the database to the same schema as it was during the development of that version (or commit).

2. Both upgrades and downgrades can be performed gracefully by the developer, making the process uniform.

3. Contains incremental changes and not the whole schema.

4. Versioning schema changes along with the code.

How do you migrate MongoDB?

I knew Django had a feature where it generated the migration files using declarative commands and I searched if MongoDB library Pymongo provided any such feature built-in, but it did not. I checked if other libraries that provided the migrations existed. There were a few, but when I inspected the code, most of them did provide a declarative interface like Django. Thus I decided to write my own. Also, I wanted to develop something open-source (MIT licensed) for a long time and this tool felt like the right one.

Mongo-Migrate

Over the weekend, I created mongo-migrate, a Python library capable of giving the developer the power to quickly create migrations, and upgrade or downgrade schema based on these incremental migration files. For most of the scenarios, you can use the command line tool mongo-migrate after you pip install, but for advanced use cases, you can import the MigrationManager class and the already created migrations.

mongo-migrate gives you three commands - create, upgrade, and downgrade. If you are eager to try it out without going through the commands, go ahead and run this command on the command line to install the package.

pip install mongo-migrate

The Create Command

It accepts a migration folder path and creates empty migration files in this folder.

usage: mongo-migrate create [-h] [--host HOST] [--port PORT] 
            [--database DATABASE] [--migrations MIGRATIONS] 
            [--title TITLE] --message MESSAGE

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           the database host
  --port PORT           the database port
  --database DATABASE   provide the database name
  --migrations MIGRATIONS
                        provide the folder to store migrations. 
                        By default creates migrations/
  --title TITLE         title will be used in the file name
  --message MESSAGE     message will be recorded as a comment 
                        inside the migration file

The template migration file (sample below) has three public methods:

1. upgrade(): Add code to upgrade the database from the current version to the target version should go here. For example, adding a collection or adding a field.

2. downgrade(): Add code relevant for downgrading the database from the current version to the target version should go here. For example, removing a collection or removing a field.

3. comment(): Adds a user-given message as a comment. Currently stored for documentation purposes.

You should add your code to perform the schema migration. (See sample below)

The Upgrade Command

The Upgrade command accepts a migration folder path and target version to go to. It systematically performs the upgrade of the database from the current version.

usage: mongo-migrate upgrade [-h] [--host HOST] [--port PORT] 
                [--database DATABASE] [--migrations MIGRATIONS] 
                [--upto UPTO]

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           the database host
  --port PORT           the database port
  --database DATABASE   provide the database name
  --migrations MIGRATIONS
                        provide the folder where the migrations are stored 
                        By default looks for migrations/
  --upto UPTO           target migration timestamp

The Downgrade Command

The Downgrade command accepts a migration folder path and target version to go to. It systematically performs the rollback of the database from the current version until reaches the target version.

usage: mongo-migrate downgrade [-h] [--host HOST] [--port PORT] 
                [--database DATABASE] [--migrations MIGRATIONS] 
                [--upto UPTO]

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           the database host
  --port PORT           the database port
  --database DATABASE   provide the database name
  --migrations MIGRATIONS
                        provide the folder where the migrations are stored. 
                        By default looks for migrations/
  --upto UPTO           target migration timestamp

How do I use mongo-migrate in a project?

Let me walk you through the steps quickly without making it sound complicated.

Step 1: Set up your virtual environment and install mongo-migrate.

Step 2: Use the mongo-migrate create command to create the first migration file template. You should provide the MongoDB host, port, and database.

Step 3: Modify the migration file created and add the code necessary to upgrade the database. Add this code to the upgrade() method.

Step 4: Also, add the steps to revert the above change in the downgrade() method.

Step 5: Run the mongo-migrate upgrade command to implement these changes onto the database.

Step 6: In case you need to revert the changes, run the mongo-migrate downgrade command.

Package Dependencies

This library was written with minimal dependency to keep it clean. The only library it uses additionally is the Pymongo library.

Conclusion

Remember, MongoDB schema migrations don't have to be complex. With mongo-migrate, managing schema changes, version control, and performing migrations is a breeze. Explore our tool, suggest ideas to improve this tool further, and simplify schema migrations!

Technical Debt is the by-product of a dysfunctional software development process.

In this article, I will try to summarise the red flags to look out for to identify if you are accumulating technical debt. For folks, who have not read the previous blog on technical debt, check it out here.

Red Flags

• Requirements or the scope of work are still not clearly defined.
  Your project owner introduced you to a new project and after the initial discussion asks you to design the software architecture, but the scope of the work is not clearly defined. This lack of clarity is going to create a sub-optimal design and it is accruing technical debt. This process is acceptable to get a jump start on a project or to create a proof-of-concept, but sooner or later you will need to refactor the code.

• Tightly coupled components - too many dependencies between functions, and functions are not modular.

  Your software architecture is too dense and it is difficult to isolate the responsibilities of different modules and functions. Whenever you are asked to implement a new feature, you end up modifying a whole lot of code because of this dependency. This is a clear indication that your codebase is due for refactoring.

• Lack of testing or test suites - Manual testing can be time-consuming and can be prone to human errors.
  When you are part of an organization (or startup) that does not have a dedicated testing team, it often falls on the team lead (or delivery head) to ensure quality software is being shipped to the end user. When the user base of the tool is within the organization, this is still manageable, but it is still a burden on the team lead to guarantee all the critical paths in the software work, and that no new software bugs are introduced. If your team doesn't believe in writing testable code or incorporating automation testing, the entire project is like a house of cards and you are headed for sleepless nights.
  It is a good point to stop and negotiate and/or educate the team on making the code testable and incorporating testing strategies into your development.

• Lack of software documentation - Nobody knows why certain things are done the way they are.
  Documentation is important, enough said. You should spend effort and time maintaining documents on software architectures, software processes, and software requirements. Requirements evolve and people working on the project change over time too. Sometimes the software processes like deployment guides, migration guides, and version control guides also must be documented to ensure the team is aware of the best practices. I would quote from the Zen of Python: Explicit is better than implicit. Ensure best practices are taught to the team and you don't expect them to just know without training.

• Lack of collaboration - knowledge is not shared around the organization and the juniors are not properly mentored.
  Amidst all the project meetings, standups, and retrospectives, we often forget to have a conversation about what interesting things you read on Hashnode or other platforms. Have open discussions with the team. When you are brainstorming strategies and requirements, encourage the team to talk. This will instill a sense of belonging and ownership in the team.

• Last-minute specification changes
  This is a pet peeve of mine. When you do not document the requirements and fail to get a sign-off, you are probably going to end up implementing something different than the person at the receiving end wanted. And sometimes, your product owner wants to wiggle in some last-minute changes. Just be aware of the technical debt accrual and remember to weed them out in a later sprint cycle.

• Poor technological leadership - when the leadership team creates frameworks that are not comprehensive or thought through.
  The software architect might have created an internal framework for the project based on the initial requirements, but if your software team is not flexible enough to refactor the framework as requirements evolve and change. You are carrying the baggage and creating workarounds.

  Another red flag to look for is when you are trying to bring improvements to the framework but the leadership constantly dismisses you. If they are not open to communication and collaboration, then it is a major red flag.

• Lack of knowledge - The developers don't know how to write elegant code.
  Elegant code, this is a subjective and touchy subject. I am not just telling about the variable naming conventions, folder structures, and Pythonic or Javascript way of doing it. If your software team is not experienced enough in that particular technology, or if they are creating tightly coupled systems. You are heading towards a scenario where you will need to refactor the code sooner or later.

Technical debt is a hidden cost that robs your business of its ability to be nimble and make needed changes for your customers. Look out for the above red flags and take action if you see your project is headed that way.

Scenario #1

You are part of a startup trying to ship the latest updates to the customers and your product owner is concerned if you take two days extra to document the code and add more test cases, the initial traction you have built with the customers will die down. You are accumulating technical debt. If a new teammate joins, this person will not have any references to learn from. He might modify code and may reintroduce bugs that were fixed before.

Scenario #2

You are part of a highly unstructured software team. The development team is involved in customer interaction and bug fixes, the team lead juggles multiple roles in the team - as the software architect, scrum master, and technical team lead. Again by not creating sustainable processes and by investing too much into one resource, you are creating a dependency. If this person leaves, or changes team, the whole knowledge is lost.

Scenario #3

If you have been part of the organization for too long and sometime in the past you and your team worked on the tool. But eventually, you have moved on and the tool is not actively maintained, and then after a year, the customer wants to add features to it. You are the only one around, but you also do not remember the requirements at that time. Also, since the tool is not maintained for a year now, the libraries are all stale. It will take as much time for you as it would take a new person to add a feature to this tool.

Technical Debt - Definition

From all the examples above, you might have gotten a fair understanding of the word "technical debt"(also known as design debt or code debt. Technical debt is the result of the software development team expediting the delivery of a certain feature or a project without proper documentation, software architecture practices, or employing resources to develop automation testing.

Is technical debt bad?

Technical debt is like monetary debt. It can be good or bad, depending on how it is used. If you use technical debt to create a quick and dirty proof of concept, it can be a valuable tool. However, if you use technical debt to create a system that is not scalable or maintainable, it can be a serious problem. Sooner or later, you will have to rewrite the entire system because it is not scalable, or, did not discuss the requirements properly.

Like borrowed money, if you do not pay it back, it accumulates interest. Similarly, if you remember to maintain the code, refactor it when necessary, and document and write test cases, then you are keeping the yin and the yang together.

How can you eliminate technical debt?

You may not be able to completely get rid of technical debt, but you and your team can reduce the ramifications of it by:

1. Allocating resources and prioritizing code documentation,

2. Paying attention to testing and test case development,

3. Refactoring the code when necessary, and,

4. Updating libraries.

I would love to hear how your team is tackling technical debt (design debt or code debt). Do let me know in the comments!