Introduction: Penetration testing & this post
Hi, we are Cob and Sebi. We are backend developers at Wanari. Our goal is to give you a general idea – without an overwhelming amount of technical detail – about:
- What drove us to start learning more about security testing
- What is a penetration test
- What kind of potential security problems can exist around a web application
- How we tested our product
- How we prioritized the issues found
- What we learned from these weeks
Instead of providing a comprehensive guide, we will give you an overall picture. It will help you to take the first steps. After that, you should definitely read more about this topic.
How it started
We developed a web application back in 2014. Our team made good choices of frameworks with security awareness. They even evaluated security threats and defined the ways to handle them.
The project became successful. A new phase began. The owner of the web application needed expansions regularly for years. There wasn’t room (available budget / free time) to keep up with the rapidly developing technologies or the changing trends of the security risks.
The time for the reevaluation this “feature-focused development” came in November 2018. For approaching a new customer, the application had to go through a security test.
So we dedicated a team to find as many potential issues as possible before the test. Our goals were to learn, practice and prepare for the actual test.
Let’s start with the basics before we continue the story!
What is a penetration test?
The informal definition of a penetration test is this: you pay a security expert (often called ethical hacker or penetration tester) to:
- Simulate a hack attack on your web application
- Simulate a hack attack on your infrastructure
- Examine company policies
- Take advantage of careless user and employee behaviors
The ethical hacker’s goal is not just to find potential weaknesses that could be used by the bad guys against your business, but s/he also attempts to exploit them. The end product of the penetration test is a list of these issues. This list contains exact technical and non-technical items, with varying severity. In case you are wondering, what kind of issues could be there, after some research we came up with three main sources of potential vulnerabilities:
What kind of potential problems could we face?
Your web app runs somewhere. It needs a server and probably databases, maybe other services (e.g. mail sender). Other web applications can run on the same server, same virtual machine or in the same environment. You may have test installations somewhere too. This means you already have a potential threat: they are possibly not as separated from each other as they are from the outside world. Developers, system administrators, sometimes support guys should have access to your web application, your databases or logs, so there are predefined ways to access any and all of these. This is hopefully not a mess, just a complicated composition of stuff. Everything has its own purpose, like in a camp.
There are houses, services, paths… The camp is under maintenance all the time. New houses are built, old ones are left unused. Your camp cannot be isolated from the outside world for doing your business. You don’t want to buy multiple camps, for facilitating multiple events at the same time. You don’t want to annoy the people with complicated controls. But you want to keep the bad guys outside.
But bad guys don’t want to stay outside, they want to:
- Come in and steal
- Move into the abandoned buildings
- Unlock lame padlocks with a hairpin
- Cut down weak padlocks with brute force
- Or just lurk, observe, and gather information for an attack
As the example shows, a real hacker has tools to discover more or less of your infrastructure. Gathering information is important for launching a fine-tuned attack on your system later, but also gives an opportunity to gain instant advantages. If one of these points is true about your infrastructure, your web application can be hacked, (even if it’s carefully developed) :
- A server or a network component is misconfigured
- A service is outdated with known critical issues
- A WordPress site is outdated with known critical issues
- Something has a default or weak password
- A production or test database is exposed to the outside world
- Another vulnerable web application or a test installation is running next to your application and is not isolated properly
- Operational services are exposed carelessly (for the sake of development convenience, for example)
Moreover, the discovery process and vulnerability detection can be automated. For example, we used scanners and automatic tools to check specific topics. We really liked these tools. We just pressed the big ‘start’ button and they could point out potential weaknesses with zero or little configuration. You should know that there are hackers scanning the internet for these low hanging fruits to then use them for their own advantage. Even if your business is not valuable for them, your infrastructure can be a useful part of their collection. If you don’t have a policy for regularly checking and keeping your infrastructure up-to-date, we’re 100% sure, you’ll be a victim sooner or later. Even a newbie can utilize these weaknesses.
The web app itself
Let us continue with our metaphor. Your web application is like a robot receptionist at your camp. (Not an AI receptionist, just a robot.) You like it, you need it. It works 24/7, it’s precise, it doesn’t get bored of repetitive administrative work. On the other hand, it just follows a predefined rule set, without thinking. Guest interactions are like this:
Robot: “Hello, who are you?”
Guest: “I’m X.Y.”
Robot: “Your name, birth date, and your mother’s name please!”
Guest: “Blah – blah…”
Robot: “What can I do for you?”
Guest: “I’d like for you to extend my stay”
Robot: “Please fill this and this form.”
The guest gets the forms, sits down, fills them, brings them back.
Robot: “A moment please.” (uses its terminal for checking the database, goes back to the drawers checks something, uses the terminal for updating the central database…) “Done Sir. Can I do anything else for you?”
The guest is your user here. The speech (and the form) is your graphical user interface and the robot is your application. The robot can interact with other systems, databases and other sources of information, depending on its task.
How could this model be cheated? Well, the only limit is our imagination! For instance, one can:
- Listen to the conversation and steel the information needed for the identification
- Insert an extra form (filled out), into the user’s pack, while the user is at the restroom
- Convince the robot that they are someone else with stolen information
- Convince the robot without the information needed for the identification, that they are someone
- Try to surprise the robot with unexpected inputs, and see what comes out of it
- Trick the robot to reveal data from the database
- Access the database, while the robot is at the drawers
- Block the front door so no one can go inside
- Even seek valuable documents in the rubbish bin
- Pay for being a real guest, do tricks from the inside, gather data
Just like with the robot, someone can misuse your application in different ways, using implementation errors or feature interference. In order to find such issues, we decided to review the code and the configuration of our application. Maybe a real pen tester doesn’t use this method as extensively as we did, but we are developers. Reading the source code and finding bad patterns was one of the most profitable methods for us. As an extra, we could get rid of unused features. This is good, since unused code could contain predefined ways into your system, yet no one’s using them anymore (or not supposed to use them at least). Clearing these features or at least making sure all the unnecessary doors are closed may get rid of dozens of potential problems.
People around the camp (guests, human receptions, maintenance staff…) are also important parts of the picture. Guests are important for your income, but they have expectations: having rest and fun. Your employees can solve complex tasks (the ones that are not simple or frequent enough to automate), but they can be tired, careless, undisciplined or can make irrational decisions under pressure.
Hackers can exploit human nature in several ways. Some of the examples include (out of the probably endless list of possibilities):
- Convince a guest to go to a different, fake office and use his/her naive gullibility
- Trick a guest into going to a different, fake office and use his/her gullibility
- Trick the guest into going to the office, identify himself and fill a posted form, without his/her awareness (it’s a bit extreme here, but not in the cyber world)
- Guess that:
- The key is under the doormat
- The PIN for the back gate is 1234
- The PIN for the back gate is the same as the PIN for the phone of an employee
- The password to the central database is on a paper next to the cash register
- Observe, that the maintenance guys leave the main building open when they are working
- Use the confusion of the new employees and convince them that the hackers are employees there too
- Operate a fake valet parking service
- Shout at the receptions on the phone, so they act as the hacker wants them to, just for peace
- Use that people are well-behaved, helpful, sympathetic and benign
- Connect on the office’s WIFI network or plug a cable into an unguarded connector port
- Create a free WIFI network at the camp (disqualify the original one) and enjoy stealing information from both the guests and the employees
You see, there is a huge number of potential problems. Are the bad guys actually going to try all of these just to hack your system? Let’s try to think like the bad guys for a moment!
It’s nearly impossible to protect an application 100%. Even if it would be possible, it would cost a lot of effort and resources. So instead of investing too much into security, let’s see how to lower the risk of an attack since it can be cheaper sometimes, especially if you think ahead.
Securing an app does costs resources, but an attack does as well. There’s the time that hackers need to invest. This (depending on the security measures of the app) can take from just several minutes up to several months or even years, all just to manage a successful attack. In some cases, expensive equipment also has to be used. So we can assume, hackers won’t attack your system if the rewards are too low compared to the costs. Increasing costs means making the system more secure. Lowering the potential reward means cutting the motivators. One of the biggest of them is data, especially the sensitive kind!
Sensitive data handling
Why you should think twice before you request or store any data? Since any sensitive or semi-sensitive data is a big motivation for the bad guys, handling and storing this data requires care, where care means extra work:
- Deciding what is sensitive according to you
- Deciding what is sensitive according to your users (the difference can affect your reputation)
- Deciding what is sensitive according to the technical guys
- Deciding what is sensitive according to legal documents (European Commission’s website, GDPR, etc…)
- Defining application-specific sensitive data. Even the fact that someone is using your system can be confidential (e.g. dating site + blackmailing the married ones)
- Deciding how sensitive it is
- Making sure stored sensitive data is properly encrypted
- Also making sure that sensitive data is encrypted while being handled or transferred
- Sensitive data requires stricter business rules inside the application, the development and the operation too
Back to the metaphor, imagine situation one: You are at your desk, writing an email to your guests to apologize. An ex-employee stole the hard drive of your database as revenge. He published its contents. The names and food preferences (non-sensitive data) of your guests are leaked. Your mood is not the best, but okayish.
Imagine situation two: Due to the renovation works, a security box (encrypted passwords) was unprotected. Someone stole it. It contained 10 of your guests car keys. The technical consultant informed you it takes a day to open the box without the key. The key is in your pocket. You and your employees are running around to find every affected guest and put their cars to safety. What a hard day, isn’t it? Fortunately, you can protect all the cars.
Imagine situation three: Same as before, but instead of a safety box, the car keys are stored in a paper bag (a.k.a. passwords without proper encryption). The thief could immediately use a key. He is in a Porsche right know, a few hundred miles away. The damage is immediate and large. You are shocked. The compensation threatens the financial stability of your business.
So overall, the more sensitive the data you store, the more you’ll have to invest into security to not end up in a situation like the last one. On the bright side:
- Properly handled data can save you or give you time to react to an unpleasant situation
- In many cases “properly handled data” may mean that it’s not being stored at all
- Proper encryption is supported by modern frameworks and third-party software components
- A conscious software feature planning and a reduced amount of sensitive data is often a great option
- Third-party services can be integrated for online paying, identifying users… This way, you can focus on your business features and may not have to solve these problems yourself.
To see the whole picture ourselves, we have decided to do a database review: what is stored, why and how. It appeared, that the application has evolved a lot over the years. New features were implemented. Others became outdated or unused and some redundant data was left in the database. But not only the app, computers, and hackers also evolved over time. One of the aftermaths is that some of the encryptions used in the database were not considered strong enough anymore.
After the testing was finished, we had to process the results. We presented our list to our CTO and other developers. Several questions and thoughts popped up at that meeting:
- How to prioritize the list?
- When two seemingly low priority issues can be combined by the attacker, does it raise any of the priorities?
- Is it a problem, when a newbie can hack into our website? (definitely YES!)
- Is it a problem, when an experienced hacker could do this, working a month on it?
- Is it a problem, when a hacker team with supercomputers can do this?
- It takes a lot of effort to fix this issue. Is it worth the effort?
- What if the customer would rather spend the money on feature development?
- Someone said, that this and these issues are not big things. Are we sure about this in our specific situation?
- We discussed the importance of the issues that are dependent on the specific kind of application we have at hand
- We discussed the importance of the issues that are highly dependent on the size and traits of the user base
- What should be considered a developer’s mistake?
- What should be considered a missing feature?
- What to do with a problem, that appeared over time (e.g. outdated dependencies, that weren’t a problem some years ago)?
Problems were different. Some were very easy to fix, others were even hard to find a solution for. Some were just minor, not really interesting by themselves, but we had to keep in mind, that the combination of just two seemingly small vulnerabilities could cause much more serious problems. We also managed to find some app-specific missing features. For example, we found out that the password complexity checker could be improved. The customer said, he didn’t want a better one, because he has only 3 administrators, who are well-trained to use strong passwords.
In the end, inspired by the OWASP Risk Rating Methodology, we sorted the problems we found into 3 groups:
- Won’t fix at all – really low priority issues (non-critical problems / hard-to-fix)
- It’s a missing feature – will be communicated to the customer
- We should fix it immediately – mistakes / ugly solution / easy-to-fix / important problems
This period was an interesting journey for us. We had a lot of fun, but sometimes the number of questions to answer was a bit frustrating. The joy – e.g. trying to break through a defense line – was incredible, but sometimes we missed our usual everyday coding tasks.
We learned, that:
- Delete, close and restrict is often the best strategy – the question “do we still need it?” can spare you a lot of time.
- Ask the people, who are working on the project! In our case, we’ve got several hints and even exact issues.
- Old software components/frameworks are painful! They pulled us back in several aspects: these generally tended to have more known issues to check and less built-in support for easily implementing the newer security features. We faced the question multiple times: accept the problem, work on a workaround or spend a lot of time on updating “everything” now and test the whole application.
- Time flies. Your business grows, new trends come in hacking, security issues are published about a component you use, etc. What was okay in 2014 (or last summer), is not necessarily good enough now. What we learned through these weeks is really useful, but will be partially outdated soon. Having an up-to-date picture and keeping the web app, the infrastructure (and more!) … at the desired level is a continuous effort.
- Security is the responsibility of developers, operators and business people too.
- Doing an in-house penetration test is fun and useful both for the developer, the company and the app itself.
- If no-one knows about something, the product will probably lack protection against it. If no one understands in your company, what is e.g. Clickjacking, your product won’t be protected against Clickjacking. If people understand SQL injection, your product will be more likely protected against SQL injection.
- Finding patterns in your potential threats can be revealing of your entire application. Analyzing these patterns may result in good prevention strategies in the future.
Now we are waiting for the results of the official test impatiently. What will be found? Will it be a bigger or smaller list, than ours? Or simply just different?
Have you had any experiences with security concerns or penetration testing? What did you find? We are truly interested in knowledge-sharing.
If you enjoyed this post, so will your friends and colleagues, share it!
Wanari is a 19-year-old custom software develoment company. Our approach to engineering is learning synchronously with the development of new technologies, so we can recommend the the most up-to-date and secure-to-use solutions to our clients. Want to learn more? Check out our website!