What's on my server?

When I first decided to run a server I had originally started with Unraid but after about a 2 months of use I was not quite happy with how it worked especially with how backups were made. I ended up starting again and finally ended up with Proxmox. What started off as a steep learning curve has now become rather trivial to manage. (Bearing in mind I broke it 3 times and had to re-install it from scratch each time)

The Basics

What I love the most about this software is the ease of use that allows me to create restore points when ever I wanted. When I start to build something new I usually first create a snapshot that allows me to start from scratch without having to build a new container / VM.

Usually once I've finished the installation and it all works as I want, I would first create a backup to my NAS (more on this later) and additionally create a snapshot called "reset". This is especially useful when you have a new case and can revert the instance to its original state to prevent different client data getting mixed up.

Backups

If you are running a Proxmox server I highly recommend having a NAS to store backups as if you are like me and have the propensity to break stuff, its easy getting back up and running

I suppose you could also chose to back things up to a physical disk but having it stored in a NAS ensures redundancy incase of disk failure.

Networking

As my motherboard has two network ports, I used my switch & Firewalla to create a separate VLAN for the second network which hosts my malware lab which I'll go into more detail later in this post.

It's essential to understand that Firewalla has a limitation when it comes to controlling traffic between devices on the same subnet (or VLAN). However, this isn't a restriction of the Firewalla product itself, but rather a fundamental aspect of how computer networks work. When devices are connected to the same subnet, they can communicate with each other directly without needing to pass through a gateway or router. This is because they share the same IP address range and are essentially 'neighbors' on the network. As a result, Firewalla (or any other firewall device) cannot create rules to block or allow traffic between devices within the same subnet, as this traffic never passes through the firewall in the first place. The devices simply talk to each other directly, bypassing the firewall altogether.

What that essentially means is that if any of my hosts in my Proxmox server were to get infected, an attacker would have the ability to access all the other machines. To get around this issue I used Proxmox's built in firewall to create a rule to block outbound access on the same subnet

As you can see above the first line is to allow all inbound traffic then followed by allowing traffic to the gateway so that it still has connectivity and lastly dropping all traffic to the subnet. Doing a test with nmap before and after confirms that my other hosts are no longer reachable from individual machines.

Now for the fun stuff

What I love about Proxmox is how easy it is to run containers. The first 4 machines (100-103) are templates I created which I installed docker on. When ever I have a new little project I want to tinker with i'll just make a clone of the template, name it, choose a disk and off I go.

So how do I access most of these machines? Well on a mac theres a nice software called Jump Desktop which allows me to save various machines RDP details. Its literally just a simple click to access a machine on my network. This works when i'm away from home too as I would just connect back to my network using wireguard first

Favourite Picks

I wont go into detail of every single machine i'm running but ill highlight some of the ones I find invaluable and use on a regular basis.

KASM

This goes to the top of my list and is probably my most used tool on my server. When ever I have something to investigate I can start any of the above containers and interact with it in a secure environment which I can then just delete and start fresh when i'm done. I also route this machine through a VPN so it hides my real IP and can easily change locations if required.

Elastic / SOF-ELK

When I had first joined incident response I conducted my investigations using a traditional windows VM but thanks to the expertise and foresight of my colleagues in my team I have since moved to using Elastic and SOF-ELK which is superior for investigations.

Velociraptor

This is a staple in every DFIR professional's toolkit. If you are interested in getting into Incident Response, this is one you need to get familiar with.

Whonix

When ever there is some data leak on the dark web by "insert threat actors name" I would use this VM to download the data usually through wget. Downloading things through tor can be cumbersome as the speeds are generally slow and the connection is unstable. Having a machine thats constantly on helps when downloading large volumes of data and I usually run the command and check on it every so often to see when (*if) its completed. If I needed to just quickly check something on tor you could also just use KASM's Tor-Browser which would be far simpler.

EDR Agents

For individuals already working in the Cybersecurity industry and have access to an EDR I recommend hosting a windows machine with an EDR agent installed. In my particular case I fortunately have access to both CrowdStrike and Microsoft's Defender for Endpoint which I have both installed separately on two Windows Server hosts. These hosts are used to trigger malware in which I then analyse the alerts and telemetry they generate.

Malware Lab

This malware lab setup came about after doing the "Practical Malware Analysis & Triage" course at TCM Security which I cant recommend enough. The setup in the course was meant to be done locally but I altered it so that it was hosted on Proxmox instead. The only downside using this method is when you start up wireshark it generates some unnecessary traffic due to the RDP connection to the host. Essentially theres a bit more filtering to do be done which I rather not do but its not a deal breaker.

To ensure safe operation, I've isolated these two machines on my network by implementing a default 'block all' rule, with a single exception: a one-way outbound allow rule that permits communication only from my Mac to the malware lab.

In terms of using the malware lab, I sometimes prefer to allow internet traffic on the FlareVM or I could also choose to simulate internet traffic by using Remnux's INetSim which should be the preferred way to analyse malware.

So how often do I use this lab? Honestly not as often as you would think. With the use of a traditional sandbox and a VM with an EDR agent installed i'm able to quickly gain a glimpse into what the malware is doing. If I need to dig a little deeper then ill make use of this lab.

AI Lab

So why did I want to build my own AI Lab when I could quite easily use solutions like ChatGPT or Grok for example? Well I have used them in the past but always found myself having to anonymise data before pasting things in which I grew tired off. I wanted a no holds barred solution that I could work with on a daily basis and incorporate into my investigative work flow. Aside from the initial investment which is most definitely high (I bought things mostly second hand to reduce the cost), the actual cost of running it is more or less the same or cheaper than the cost of signing up for a paid service from any of the big vendors currently.

How does it all work?

Well the most important part is ensuring that you passthrough your GPU(s) to a VM which can only be attached to one host at a time. From there, make sure you have the proper Nvidia drivers installed and working. In my particular case as I built it on Kali Linux I used this guide. The primary reason for me wanting to use the Kali OS was that I had also wanted the AI Lab to also double as a password cracking machine which is favourable with this kind of setup.

AI Tools

In terms of tooling i'll show you some of the things i'm currently running in my lab.

Ollama / Open WebUI

The core of the lab is built upon these two tools working together which allow me to run multiple models. This is especially helpful as I could ask the same question to multiple models to confirm how accurate the answers are and rule out potential hallucinations. Essentially with 48GB of combined memory the rig is able to handle models up to 70B quite easily.

One thing I also found very annoying with using commercialized AI solutions were the fact that they were rather restrictive in nature. Asking questions about decoding malware for example would often lead to rejections saying its not feasible to do that for what ever reason. Obviously you could trick this by saying you are a security researcher for example but its just become a little tedious. Its far easier finding an unrestricted model on the Ollama library that could do the job with no questions asked.

Perplexica (Perplexity Clone)

Perplexica appears to be inspired by or a clone of Perplexity which is an innovative AI-powered search engine which in layman's terms consolidates information from various sources and presents the data in a simple easily digestible format.

As you can see a simple question was answered in a concise format and whats most important is it provides links to the sources where it found the information from so you can determine for yourself how relevant or true that information is. I have had great success with using this tool by asking questions related to a newly released vulnerability and understanding it in greater detail. I particularly like being able to ask follow up questions to really gain a grasp on a topic quickly. If you don't have the resources to run this locally you can try out Perplexity for yourself. Ironically after setting up my AI Lab I received a 1 year trial period through my mobile provider. I should do some tests to compare the two but the benefits of perplexica is that you can again customise the chat models and also embedding models which would vary the output.

RagFlow

This powerful tool holds great promise for interacting with documents in a more efficient and centralized manner. The core idea behind this tool is to create a comprehensive repository of documents, which can then be queried from a single point of access. This approach has the potential to revolutionize the way I work with documents, enabling me to quickly locate specific information, analyze data, and gain valuable insights.

It goes without saying that using a AI to query private and sensitive documents should be done with caution. This is where the power of self hosting your own AI lab really starts to shine.

Page Assist

This tool has similar capabilities as Perplexica in which I could use it as a search engine however what I usually use it for is to interact with specific web pages which may have a lot of information

Closing Notes

Congrats if you made it this far and hopefully its given those of you wanting to embark on a similar endeavor some helpful insights. As always, i'm available on LinkedIn for any further questions.